[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:15 dist-test-slave-k5rg krb5kdc[13024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:15 dist-test-slave-k5rg krb5kdc[13024](info): set up 2 sockets
Sep 28 02:58:15 dist-test-slave-k5rg krb5kdc[13024](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:58:16.125571 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.001s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:58:16 dist-test-slave-k5rg krb5kdc[13024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:58:16.286425 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41583
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:41583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:16.582005 13037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:16.582266 13037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:16.582396 13037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:16.589382 13037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:16.589481 13037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:16.589548 13037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:16.589643 13037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:16.589704 13037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:16.599205 13037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:41583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:16.600641 13037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:16.601926 13037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:16.612720 13042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:16.612681 13043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:16.613114 13045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:16.613713 13037 server_base.cc:1034] running on GCE node
I20260928 02:58:16.614446 13037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:16.615780 13037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:16.616997 13037 hybrid_clock.cc:648] HybridClock initialized: now 1790564296616939 us; error 46 us; skew 500 ppm
I20260928 02:58:16.617334 13037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:16 dist-test-slave-k5rg krb5kdc[13024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564296, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:16.620332 13037 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:58:16.621877 13037 webserver.cc:460] Webserver started at http://127.12.182.126:37255/ using document root <none> and password file <none>
I20260928 02:58:16.622543 13037 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:16.622656 13037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:16.622946 13037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:16.625872 13037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "ddc8cf21e386461196cffb7448727863"
format_stamp: "Formatted at 2026-09-28 02:58:16 on dist-test-slave-k5rg"
server_key: "dc296da53fd79431435d8aa48a3c2963"
server_key_iv: "411b2dcc5543aa622282f397bca58508"
server_key_version: "encryptionkey@0"
I20260928 02:58:16.626579 13037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "ddc8cf21e386461196cffb7448727863"
format_stamp: "Formatted at 2026-09-28 02:58:16 on dist-test-slave-k5rg"
server_key: "dc296da53fd79431435d8aa48a3c2963"
server_key_iv: "411b2dcc5543aa622282f397bca58508"
server_key_version: "encryptionkey@0"
I20260928 02:58:16.631464 13037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:58:16.634734 13052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:16.636003 13037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:58:16.636155 13037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "ddc8cf21e386461196cffb7448727863"
format_stamp: "Formatted at 2026-09-28 02:58:16 on dist-test-slave-k5rg"
server_key: "dc296da53fd79431435d8aa48a3c2963"
server_key_iv: "411b2dcc5543aa622282f397bca58508"
server_key_version: "encryptionkey@0"
I20260928 02:58:16.636318 13037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:16.661916 13037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:16.665997 13037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:16.666246 13037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:16.683233 13037 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:41583
I20260928 02:58:16.683261 13103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:41583 every 8 connection(s)
I20260928 02:58:16.684516 13037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:16.688120 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:16.690773 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 13037
I20260928 02:58:16.691040 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:16.692847 13017 external_mini_cluster.cc:1428] Setting key f603478f15fdbe1b6977a08ea0160349
I20260928 02:58:16.699122 13104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863: Bootstrap starting.
I20260928 02:58:16.701925 13104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:16.702822 13104 log.cc:824] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:16.705322 13104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863: No bootstrap required, opened a new log
I20260928 02:58:16.712646 13104 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc8cf21e386461196cffb7448727863" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41583 } }
I20260928 02:58:16.712972 13104 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:16.713073 13104 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc8cf21e386461196cffb7448727863, State: Initialized, Role: FOLLOWER
I20260928 02:58:16.713742 13104 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc8cf21e386461196cffb7448727863" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41583 } }
I20260928 02:58:16.714030 13104 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:16.714169 13104 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:16.714360 13104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:16.717417 13104 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc8cf21e386461196cffb7448727863" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41583 } }
I20260928 02:58:16.717913 13104 leader_election.cc:302] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddc8cf21e386461196cffb7448727863; no voters:
I20260928 02:58:16.718509 13104 leader_election.cc:288] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:16.718602 13109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 02:58:16 dist-test-slave-k5rg krb5kdc[13024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:16.719743 13109 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [term 1 LEADER]: Becoming Leader. State: Replica: ddc8cf21e386461196cffb7448727863, State: Running, Role: LEADER
I20260928 02:58:16.720235 13109 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc8cf21e386461196cffb7448727863" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41583 } }
I20260928 02:58:16.720794 13104 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:16.722951 13110 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddc8cf21e386461196cffb7448727863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc8cf21e386461196cffb7448727863" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41583 } } }
I20260928 02:58:16.723326 13110 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:16.724495 13111 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddc8cf21e386461196cffb7448727863. Latest consensus state: current_term: 1 leader_uuid: "ddc8cf21e386461196cffb7448727863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc8cf21e386461196cffb7448727863" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41583 } } }
I20260928 02:58:16.724893 13111 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:16.729439 13114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:16.731324 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:16.696559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.12.182.126:41583)
0928 02:58:16.697326 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:16.697343 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:16.698507 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 02:58:16.706877 (+ 8370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:16.706897 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:16.707249 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:16.707579 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:16.710240 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:16.711533 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:16.713086 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:16.714106 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:16.714290 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:16.720826 (+ 6536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:16.720836 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:16.720855 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:16.720912 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:16.725191 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:16.726389 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:16.726398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:16.726407 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:16.726641 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:16.727258 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:16.727263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:16.727267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:16.727998 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:16.729353 (+ 1355us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:16.729690 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 02:58:16.730102 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":340,"threads_started":1}
I20260928 02:58:16.736263 13114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:16.745447 13114 catalog_manager.cc:1347] Generated new cluster ID: e09ac0a8b4c7442a9c2f33b1dbad07a8
I20260928 02:58:16.745556 13114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:16.754159 13114 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:16.755867 13114 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:16.767411 13114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddc8cf21e386461196cffb7448727863: Generated new TSK 0
I20260928 02:58:16.768417 13114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:58:16.810699 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41583
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:17.112562 13130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:17.112887 13130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:17.113039 13130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:17.120172 13130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:17.120272 13130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:17.120417 13130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:58:17.130119 13130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41583
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:17.131271 13130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:17.132498 13130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:17.144799 13135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:17.144846 13136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:17.145141 13138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:17.145625 13130 server_base.cc:1034] running on GCE node
I20260928 02:58:17.146171 13130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:17.146968 13130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:17.148141 13130 hybrid_clock.cc:648] HybridClock initialized: now 1790564297148087 us; error 45 us; skew 500 ppm
I20260928 02:58:17.148412 13130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:17 dist-test-slave-k5rg krb5kdc[13024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:17.151338 13130 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:58:17.152642 13130 webserver.cc:460] Webserver started at http://127.12.182.65:34837/ using document root <none> and password file <none>
I20260928 02:58:17.153299 13130 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:17.153403 13130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:17.153734 13130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:17.156438 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "755ba54412fc49918969b2399227cf75"
format_stamp: "Formatted at 2026-09-28 02:58:17 on dist-test-slave-k5rg"
server_key: "22d35a660d42cd407302a7d95fb7fa24"
server_key_iv: "7ba1c9abc76ad62a50a3c516b074c60c"
server_key_version: "encryptionkey@0"
I20260928 02:58:17.157136 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "755ba54412fc49918969b2399227cf75"
format_stamp: "Formatted at 2026-09-28 02:58:17 on dist-test-slave-k5rg"
server_key: "22d35a660d42cd407302a7d95fb7fa24"
server_key_iv: "7ba1c9abc76ad62a50a3c516b074c60c"
server_key_version: "encryptionkey@0"
I20260928 02:58:17.161885 13130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260928 02:58:17.165338 13145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:17.166693 13130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:17.166889 13130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "755ba54412fc49918969b2399227cf75"
format_stamp: "Formatted at 2026-09-28 02:58:17 on dist-test-slave-k5rg"
server_key: "22d35a660d42cd407302a7d95fb7fa24"
server_key_iv: "7ba1c9abc76ad62a50a3c516b074c60c"
server_key_version: "encryptionkey@0"
I20260928 02:58:17.167063 13130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:17.187198 13130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:17.191145 13130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:17.191416 13130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:17.192441 13130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:17.193710 13130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:17.193790 13130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:17.193905 13130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:17.193955 13130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:17.222688 13130 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:42793
I20260928 02:58:17.222743 13257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:42793 every 8 connection(s)
I20260928 02:58:17.223891 13130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:58:17.224715 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 13130
I20260928 02:58:17.224946 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:58:17.225574 13017 external_mini_cluster.cc:1428] Setting key 08f9704c2768e76a59288df3759dd00e
Sep 28 02:58:17 dist-test-slave-k5rg krb5kdc[13024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
I20260928 02:58:17.243490 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:17.226285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:53971 (local address 127.12.182.126:41583)
0928 02:58:17.226456 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:17.226462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:17.227660 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 02:58:17.229404 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:17.229409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:17.229489 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:17.229590 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:17.231084 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:17.232122 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:17.233589 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:17.234160 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:17.234268 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:17.237664 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:17.237670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:17.237676 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:17.237718 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:17.240512 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:17.241219 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:17.241224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:17.241228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:17.241345 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:17.241799 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:17.241804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:17.241807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:17.242152 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:17.242283 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:17.242946 (+ 663us) server_negotiation.cc:299] Negotiation successful
0928 02:58:17.243159 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:58:17.244474 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:17.226747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41583 (local address 127.12.182.65:53971)
0928 02:58:17.227508 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:17.227557 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:17.229159 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:17.229807 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:17.229815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:17.230288 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:17.230915 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:17.230939 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:17.232268 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:17.232272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:17.233465 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:17.233477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:17.234348 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:17.234353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:17.234559 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:17.235184 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:17.235209 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:17.237421 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:17.240657 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:17.240665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:17.240682 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:17.241109 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:17.241449 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:17.241453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:17.241457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:17.241697 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:17.242267 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:17.242274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:17.242598 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 02:58:17.242833 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 02:58:17.243355 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":303,"threads_started":1}
I20260928 02:58:17.245952 13258 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41583
I20260928 02:58:17.246229 13258 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:17.246897 13258 heartbeater.cc:511] Master 127.12.182.126:41583 requested a full tablet report, sending...
I20260928 02:58:17.248776 13069 ts_manager.cc:194] Registered new tserver with Master: 755ba54412fc49918969b2399227cf75 (127.12.182.65:42793)
I20260928 02:58:17.250208 13069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:53971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:58:17.260295 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41583
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:17.555527 13263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:17.555802 13263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:17.555903 13263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:17.563050 13263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:17.563164 13263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:17.563278 13263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:58:17.573057 13263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41583
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:17.574288 13263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:17.575531 13263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:17.588373 13268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:17.588336 13269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:17.588407 13271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:17.589341 13263 server_base.cc:1034] running on GCE node
I20260928 02:58:17.589960 13263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:17.590787 13263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:17.592005 13263 hybrid_clock.cc:648] HybridClock initialized: now 1790564297591951 us; error 46 us; skew 500 ppm
I20260928 02:58:17.592315 13263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:17 dist-test-slave-k5rg krb5kdc[13024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:17.595299 13263 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:58:17.596580 13263 webserver.cc:460] Webserver started at http://127.12.182.66:38235/ using document root <none> and password file <none>
I20260928 02:58:17.597213 13263 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:17.597323 13263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:17.597621 13263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:17.600395 13263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "b1d61da65baa48a38a37f5d4320f62c7"
format_stamp: "Formatted at 2026-09-28 02:58:17 on dist-test-slave-k5rg"
server_key: "166d8e4eb1ee2717920af83f1468763b"
server_key_iv: "d9cd53bffc421a72b00e614ff8d3f0d4"
server_key_version: "encryptionkey@0"
I20260928 02:58:17.601130 13263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "b1d61da65baa48a38a37f5d4320f62c7"
format_stamp: "Formatted at 2026-09-28 02:58:17 on dist-test-slave-k5rg"
server_key: "166d8e4eb1ee2717920af83f1468763b"
server_key_iv: "d9cd53bffc421a72b00e614ff8d3f0d4"
server_key_version: "encryptionkey@0"
I20260928 02:58:17.606000 13263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 02:58:17.609333 13278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:17.610637 13263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:58:17.610777 13263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "b1d61da65baa48a38a37f5d4320f62c7"
format_stamp: "Formatted at 2026-09-28 02:58:17 on dist-test-slave-k5rg"
server_key: "166d8e4eb1ee2717920af83f1468763b"
server_key_iv: "d9cd53bffc421a72b00e614ff8d3f0d4"
server_key_version: "encryptionkey@0"
I20260928 02:58:17.610970 13263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:17.631793 13263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:17.635597 13263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:17.635848 13263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:17.636816 13263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:17.638187 13263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:17.638265 13263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:17.638381 13263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:17.638432 13263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:17.667973 13263 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:37239
I20260928 02:58:17.668016 13390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:37239 every 8 connection(s)
I20260928 02:58:17.669222 13263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:58:17.674449 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 13263
I20260928 02:58:17.674639 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:58:17.675146 13017 external_mini_cluster.cc:1428] Setting key 3c47a4649bc40d3db820d2153e425c11
Sep 28 02:58:17 dist-test-slave-k5rg krb5kdc[13024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
I20260928 02:58:17.691267 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:17.671811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:57847 (local address 127.12.182.126:41583)
0928 02:58:17.671981 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:17.671985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:17.673119 (+ 1134us) server_negotiation.cc:407] Connection header received
0928 02:58:17.675424 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:17.675428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:17.675502 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:17.675586 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:17.677277 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:17.678029 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:17.680146 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:17.680550 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:17.680651 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:17.684563 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:17.684569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:17.684574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:17.684611 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:17.687967 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:17.688830 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:17.688836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:17.688840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:17.689014 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:17.689454 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:17.689459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:17.689463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:17.689864 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:17.690035 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:17.690837 (+ 802us) server_negotiation.cc:299] Negotiation successful
0928 02:58:17.691023 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:58:17.692571 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:17.672321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41583 (local address 127.12.182.66:57847)
0928 02:58:17.672974 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:17.673010 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:17.675108 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:17.675835 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:17.675844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:17.676433 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:17.677096 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:17.677127 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:17.678184 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:17.678189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:17.680001 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:17.680016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:17.680761 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:17.680767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:17.681040 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:17.681776 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:17.681797 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:17.684324 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:17.688117 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:17.688126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:17.688141 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:17.688701 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:17.689118 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:17.689123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:17.689127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:17.689342 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:17.690027 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:17.690035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:17.690430 (+ 395us) client_negotiation.cc:769] Sending connection context
0928 02:58:17.690703 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 02:58:17.691216 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":241,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
I20260928 02:58:17.693996 13391 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41583
I20260928 02:58:17.694281 13391 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:17.694964 13391 heartbeater.cc:511] Master 127.12.182.126:41583 requested a full tablet report, sending...
I20260928 02:58:17.696307 13069 ts_manager.cc:194] Registered new tserver with Master: b1d61da65baa48a38a37f5d4320f62c7 (127.12.182.66:37239)
I20260928 02:58:17.697103 13069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:57847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:58:17.713388 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41583
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:18.013146 13396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:18.013428 13396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:18.013569 13396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:18.020720 13396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:18.020828 13396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:18.020987 13396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:58:18.030470 13396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41583
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:18.031677 13396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:18.032918 13396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:18.045346 13401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:18.045583 13404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:18.045536 13402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:18.046434 13396 server_base.cc:1034] running on GCE node
I20260928 02:58:18.046986 13396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:18.047819 13396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:18.049012 13396 hybrid_clock.cc:648] HybridClock initialized: now 1790564298048964 us; error 42 us; skew 500 ppm
I20260928 02:58:18.049304 13396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:18.052207 13396 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:58:18.053531 13396 webserver.cc:460] Webserver started at http://127.12.182.67:35789/ using document root <none> and password file <none>
I20260928 02:58:18.054164 13396 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:18.054258 13396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:18.054502 13396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:18.057353 13396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "18a280e43df340fba1d05a583fb3fd8e"
format_stamp: "Formatted at 2026-09-28 02:58:18 on dist-test-slave-k5rg"
server_key: "fe4f58b46e530ffdd18e7a3072fa563a"
server_key_iv: "b31ede44823904851c47f59547012874"
server_key_version: "encryptionkey@0"
I20260928 02:58:18.058126 13396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "18a280e43df340fba1d05a583fb3fd8e"
format_stamp: "Formatted at 2026-09-28 02:58:18 on dist-test-slave-k5rg"
server_key: "fe4f58b46e530ffdd18e7a3072fa563a"
server_key_iv: "b31ede44823904851c47f59547012874"
server_key_version: "encryptionkey@0"
I20260928 02:58:18.062917 13396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 02:58:18.066308 13411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:18.067579 13396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:18.067724 13396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "18a280e43df340fba1d05a583fb3fd8e"
format_stamp: "Formatted at 2026-09-28 02:58:18 on dist-test-slave-k5rg"
server_key: "fe4f58b46e530ffdd18e7a3072fa563a"
server_key_iv: "b31ede44823904851c47f59547012874"
server_key_version: "encryptionkey@0"
I20260928 02:58:18.067903 13396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:18.082381 13396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:18.086148 13396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:18.086405 13396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:18.087276 13396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:18.088487 13396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:18.088554 13396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:18.088660 13396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:18.088711 13396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:18.118803 13396 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:40787
I20260928 02:58:18.118845 13523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:40787 every 8 connection(s)
I20260928 02:58:18.119998 13396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:58:18.126849 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 13396
I20260928 02:58:18.127049 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:58:18.127480 13017 external_mini_cluster.cc:1428] Setting key d465729e447925d7fba4501a58d07c10
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:18.138127 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:18.122361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:53751 (local address 127.12.182.126:41583)
0928 02:58:18.122515 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:18.122519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:18.123602 (+ 1083us) server_negotiation.cc:407] Connection header received
0928 02:58:18.125345 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:18.125351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:18.125433 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:18.125514 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:18.127116 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.127934 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.129661 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.130117 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.130214 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:18.133288 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:18.133293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:18.133302 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:18.133337 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:18.135707 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.136297 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.136301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.136303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.136410 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.136775 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.136777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.136779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.137043 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:18.137159 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:18.137719 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 02:58:18.137882 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:58:18.139104 13526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:18.122784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41583 (local address 127.12.182.67:53751)
0928 02:58:18.123452 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:18.123489 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:18.125101 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:18.125765 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:18.125772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:18.126233 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:18.126941 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:18.126967 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.128087 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.128092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:18.129525 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:18.129537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.130332 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.130339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:18.130546 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:18.131141 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:18.131161 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:18.133095 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:18.135820 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:18.135826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:18.135837 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:18.136203 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:18.136520 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:18.136523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:18.136525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:18.136697 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:18.137139 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:18.137144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:18.137422 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 02:58:18.137668 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 02:58:18.138029 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":245,"threads_started":1}
I20260928 02:58:18.140393 13524 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41583
I20260928 02:58:18.140669 13524 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:18.141444 13524 heartbeater.cc:511] Master 127.12.182.126:41583 requested a full tablet report, sending...
I20260928 02:58:18.143016 13068 ts_manager.cc:194] Registered new tserver with Master: 18a280e43df340fba1d05a583fb3fd8e (127.12.182.67:40787)
I20260928 02:58:18.143817 13068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:53751
I20260928 02:58:18.155084 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:58:18.196045 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:18.181187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.12.182.65:42793)
0928 02:58:18.181807 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:18.181818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:18.181979 (+ 161us) server_negotiation.cc:407] Connection header received
0928 02:58:18.182150 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:18.182157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:18.182384 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:18.182524 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:18.184064 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.185074 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.186325 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.186869 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.186979 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:18.190797 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:18.190808 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:18.190821 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:18.190861 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:18.193349 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.194069 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.194079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.194082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.194231 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.194670 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.194675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.194679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.195046 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:18.195270 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:18.195479 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 02:58:18.195679 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":266,"threads_started":1}
W20260928 02:58:18.197191 13173 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42386
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:58:18.237762 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:18.222625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.12.182.65:42793)
0928 02:58:18.222819 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:18.222824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:18.223109 (+ 285us) server_negotiation.cc:407] Connection header received
0928 02:58:18.223300 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:18.223304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:18.223373 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:18.223455 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:18.225045 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.225934 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.227742 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.228254 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.228355 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:18.233009 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:18.233015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:18.233018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:18.233051 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:18.235440 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.236060 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.236064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.236066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.236178 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.236471 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.236474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.236476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.236759 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:18.236930 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:18.237239 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 02:58:18.237427 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:58:18.239332 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 13130
I20260928 02:58:18.252976 13258 heartbeater.cc:503] Master 127.12.182.126:41583 was elected leader, sending a full tablet report...
I20260928 02:58:18.253947 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:18.240368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.12.182.65:42793)
0928 02:58:18.240550 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:18.240555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:18.240891 (+ 336us) server_negotiation.cc:407] Connection header received
0928 02:58:18.241046 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:18.241050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:18.241104 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:18.241176 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:18.242447 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.243461 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.245225 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.245850 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.245958 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:18.248798 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:18.248804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:18.248809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:18.248847 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:18.251458 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.252039 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.252043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.252045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.252156 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.252478 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.252482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.252484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.252800 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:18.253029 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:18.253336 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 02:58:18.253561 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:58:18.254660 13253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:18.335566 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 13130
I20260928 02:58:18.348330 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 13263
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 02:58:18.363539 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:18.349180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.12.182.66:37239)
0928 02:58:18.349764 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:18.349771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:18.349795 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:58:18.349853 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:18.349859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:18.350059 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:18.350187 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:18.351368 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.352428 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.353675 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.354265 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.354384 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:18.358231 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:18.358243 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:18.358256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:18.358294 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:18.360840 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.361472 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.361479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.361481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.361625 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.362012 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.362018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.362021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.362349 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:18.362634 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:18.362926 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:58:18.363183 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":287,"threads_started":1}
I20260928 02:58:18.364734 13386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:18.445520 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 13263
I20260928 02:58:18.457814 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 13396
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 02:58:18.473371 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:18.458735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.12.182.67:40787)
0928 02:58:18.459388 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:18.459395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:18.459435 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:58:18.459494 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:18.459500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:18.459709 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:18.459829 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:18.460945 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.462059 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.463281 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:18.463819 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:18.463927 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:18.468111 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:18.468122 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:18.468135 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:18.468166 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:18.470846 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.471509 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.471519 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.471522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.471636 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:18.471942 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:18.471946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:18.471948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:18.472254 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:18.472520 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:18.472775 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 02:58:18.473011 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":361,"threads_started":1}
I20260928 02:58:18.474602 13519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:18.553715 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 13396
I20260928 02:58:18.565848 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 13037
I20260928 02:58:18.567010 13099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:18.667513 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 13037
2026-09-28T02:58:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3553 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13545](info): set up 2 sockets
Sep 28 02:58:18 dist-test-slave-k5rg krb5kdc[13545](info): commencing operation
krb5kdc: starting...
W20260928 02:58:19.694689 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:58:19 dist-test-slave-k5rg krb5kdc[13545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:58:19.855412 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:45753
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:45753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:20.158540 13558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:20.158823 13558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:20.158921 13558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:20.166050 13558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:20.166153 13558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:20.166224 13558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:20.166302 13558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:20.166353 13558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:20.176201 13558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:45753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:20.177459 13558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:20.178704 13558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:20.190075 13563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:20.190183 13566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:20.190177 13564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:20.191067 13558 server_base.cc:1034] running on GCE node
I20260928 02:58:20.191927 13558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:20.193482 13558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:20.194695 13558 hybrid_clock.cc:648] HybridClock initialized: now 1790564300194634 us; error 57 us; skew 500 ppm
I20260928 02:58:20.195051 13558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:20 dist-test-slave-k5rg krb5kdc[13545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:20.200774 13558 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:58:20.202277 13558 webserver.cc:460] Webserver started at http://127.12.182.126:40011/ using document root <none> and password file <none>
I20260928 02:58:20.202924 13558 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:20.203022 13558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:20.203286 13558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:20.206243 13558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "7b273ef42a724f618fc7fcd874cac88d"
format_stamp: "Formatted at 2026-09-28 02:58:20 on dist-test-slave-k5rg"
server_key: "ec74a2060e1d4a7b528eb14b4b8e8655"
server_key_iv: "f9d27280d44d9bda91ec841622db97ce"
server_key_version: "encryptionkey@0"
I20260928 02:58:20.206918 13558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "7b273ef42a724f618fc7fcd874cac88d"
format_stamp: "Formatted at 2026-09-28 02:58:20 on dist-test-slave-k5rg"
server_key: "ec74a2060e1d4a7b528eb14b4b8e8655"
server_key_iv: "f9d27280d44d9bda91ec841622db97ce"
server_key_version: "encryptionkey@0"
I20260928 02:58:20.211908 13558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:58:20.215189 13573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:20.216488 13558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:58:20.216629 13558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "7b273ef42a724f618fc7fcd874cac88d"
format_stamp: "Formatted at 2026-09-28 02:58:20 on dist-test-slave-k5rg"
server_key: "ec74a2060e1d4a7b528eb14b4b8e8655"
server_key_iv: "f9d27280d44d9bda91ec841622db97ce"
server_key_version: "encryptionkey@0"
I20260928 02:58:20.216789 13558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:20.238718 13558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:20.242830 13558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:20.243065 13558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:20.260021 13558 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:45753
I20260928 02:58:20.260056 13624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:45753 every 8 connection(s)
I20260928 02:58:20.261372 13558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:20.265027 13625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:20.270097 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 13558
I20260928 02:58:20.270298 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:20.270892 13017 external_mini_cluster.cc:1428] Setting key c65e882c2437605178a49b6161a4ac7f
I20260928 02:58:20.275761 13625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d: Bootstrap starting.
I20260928 02:58:20.279299 13625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:20.280526 13625 log.cc:824] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:20.283921 13625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d: No bootstrap required, opened a new log
Sep 28 02:58:20 dist-test-slave-k5rg krb5kdc[13545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:20.291903 13625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b273ef42a724f618fc7fcd874cac88d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45753 } }
I20260928 02:58:20.292282 13625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:20.292435 13625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b273ef42a724f618fc7fcd874cac88d, State: Initialized, Role: FOLLOWER
I20260928 02:58:20.293061 13625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b273ef42a724f618fc7fcd874cac88d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45753 } }
I20260928 02:58:20.293292 13625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:20.293427 13625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:20.293597 13625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:20.294031 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:20.272793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.12.182.126:45753)
0928 02:58:20.273658 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:20.273672 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:20.273719 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:58:20.275333 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:20.275356 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:20.275765 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:20.276096 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:20.277855 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:20.279138 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:20.280791 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:20.281849 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:20.281992 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:20.286409 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:20.286418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:20.286434 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:20.286468 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:20.289772 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:20.290539 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:20.290546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:20.290556 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:20.290742 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:20.291104 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:20.291111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:20.291113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:20.291651 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:20.291910 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:20.292402 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 02:58:20.292875 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":378,"threads_started":1}
I20260928 02:58:20.296398 13625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b273ef42a724f618fc7fcd874cac88d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45753 } }
I20260928 02:58:20.296909 13625 leader_election.cc:302] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b273ef42a724f618fc7fcd874cac88d; no voters:
I20260928 02:58:20.297469 13625 leader_election.cc:288] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:20.297583 13630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:20.298211 13630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [term 1 LEADER]: Becoming Leader. State: Replica: 7b273ef42a724f618fc7fcd874cac88d, State: Running, Role: LEADER
I20260928 02:58:20.298753 13630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b273ef42a724f618fc7fcd874cac88d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45753 } }
I20260928 02:58:20.299454 13625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:20.301354 13631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b273ef42a724f618fc7fcd874cac88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b273ef42a724f618fc7fcd874cac88d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45753 } } }
I20260928 02:58:20.301756 13631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:20.302278 13632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b273ef42a724f618fc7fcd874cac88d. Latest consensus state: current_term: 1 leader_uuid: "7b273ef42a724f618fc7fcd874cac88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b273ef42a724f618fc7fcd874cac88d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45753 } } }
I20260928 02:58:20.302592 13632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:20.304028 13639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:20.308658 13639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:20.317696 13639 catalog_manager.cc:1347] Generated new cluster ID: a084070e9303410a97ee6d4f53c430d2
I20260928 02:58:20.317819 13639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:20.331677 13639 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:20.332914 13639 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:20.344951 13639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b273ef42a724f618fc7fcd874cac88d: Generated new TSK 0
I20260928 02:58:20.346058 13639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:58:20.401095 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45753
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:20.722571 13652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:20.722827 13652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:20.722918 13652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:20.729882 13652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:20.729979 13652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:20.730090 13652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:58:20.739593 13652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45753
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:20.740796 13652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:20.742065 13652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:20.754364 13658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:20.754395 13657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:20.754426 13660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:20.755193 13652 server_base.cc:1034] running on GCE node
I20260928 02:58:20.755815 13652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:20.756750 13652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:20.757939 13652 hybrid_clock.cc:648] HybridClock initialized: now 1790564300757887 us; error 52 us; skew 500 ppm
I20260928 02:58:20.758239 13652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:20 dist-test-slave-k5rg krb5kdc[13545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:20.761147 13652 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:58:20.762634 13652 webserver.cc:460] Webserver started at http://127.12.182.65:45315/ using document root <none> and password file <none>
I20260928 02:58:20.763264 13652 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:20.763362 13652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:20.763651 13652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:20.766835 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "f5c137b2aab54fd78940802511bd7516"
format_stamp: "Formatted at 2026-09-28 02:58:20 on dist-test-slave-k5rg"
server_key: "d783cd9f3f6219fc60290a723b2b565b"
server_key_iv: "35a7394a09316370ce874dcdf0371e97"
server_key_version: "encryptionkey@0"
I20260928 02:58:20.767578 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "f5c137b2aab54fd78940802511bd7516"
format_stamp: "Formatted at 2026-09-28 02:58:20 on dist-test-slave-k5rg"
server_key: "d783cd9f3f6219fc60290a723b2b565b"
server_key_iv: "35a7394a09316370ce874dcdf0371e97"
server_key_version: "encryptionkey@0"
I20260928 02:58:20.772476 13652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:58:20.775830 13667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:20.777199 13652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 02:58:20.777334 13652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "f5c137b2aab54fd78940802511bd7516"
format_stamp: "Formatted at 2026-09-28 02:58:20 on dist-test-slave-k5rg"
server_key: "d783cd9f3f6219fc60290a723b2b565b"
server_key_iv: "35a7394a09316370ce874dcdf0371e97"
server_key_version: "encryptionkey@0"
I20260928 02:58:20.777500 13652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:20.795245 13652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:20.799422 13652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:20.799687 13652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:20.800642 13652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:20.801980 13652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:20.802067 13652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:20.802187 13652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:20.802237 13652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:20.831768 13652 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:43387
I20260928 02:58:20.831835 13779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:43387 every 8 connection(s)
I20260928 02:58:20.833041 13652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:58:20.838982 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 13652
I20260928 02:58:20.839187 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:58:20.839766 13017 external_mini_cluster.cc:1428] Setting key fda9e7b5154833d64a03205811017c71
Sep 28 02:58:20 dist-test-slave-k5rg krb5kdc[13545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:20.852861 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:20.835423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:60897 (local address 127.12.182.126:45753)
0928 02:58:20.835854 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:20.835860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:20.836797 (+ 937us) server_negotiation.cc:407] Connection header received
0928 02:58:20.838581 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:20.838587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:20.838675 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:20.838808 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:20.840306 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:20.841158 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:20.842848 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:20.843449 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:20.843566 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:20.847294 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:20.847301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:20.847306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:20.847356 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:20.849771 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:20.850652 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:20.850657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:20.850660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:20.850801 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:20.851255 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:20.851257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:20.851259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:20.851512 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:20.851618 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:20.852441 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 02:58:20.852601 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":199,"threads_started":1}
I20260928 02:58:20.854391 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:20.835965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45753 (local address 127.12.182.65:60897)
0928 02:58:20.836646 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:20.836687 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:20.838289 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:20.838977 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:20.838988 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:20.839550 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:20.840135 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:20.840155 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:20.841320 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:20.841325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:20.842691 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:20.842704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:20.843656 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:20.843661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:20.843899 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:20.844646 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:20.844677 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:20.847040 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:20.849899 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:20.849908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:20.849925 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:20.850526 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:20.850902 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:20.850907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:20.850911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:20.851160 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:20.851631 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:20.851639 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:20.852082 (+ 443us) client_negotiation.cc:769] Sending connection context
0928 02:58:20.852398 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 02:58:20.852933 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":262,"threads_started":1}
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
I20260928 02:58:20.855877 13780 heartbeater.cc:347] Connected to a master server at 127.12.182.126:45753
I20260928 02:58:20.856197 13780 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:20.856938 13780 heartbeater.cc:511] Master 127.12.182.126:45753 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
I20260928 02:58:20.859285 13590 ts_manager.cc:194] Registered new tserver with Master: f5c137b2aab54fd78940802511bd7516 (127.12.182.65:43387)
I20260928 02:58:20.861104 13590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:60897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:58:20.876593 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45753
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:21.175469 13786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:21.175729 13786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:21.175822 13786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:21.182973 13786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:21.183072 13786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:21.183192 13786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:58:21.192559 13786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45753
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:21.193732 13786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:21.194972 13786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:21.206804 13791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:21.206982 13792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:21.207239 13794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:21.207774 13786 server_base.cc:1034] running on GCE node
I20260928 02:58:21.208339 13786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:21.209302 13786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:21.210501 13786 hybrid_clock.cc:648] HybridClock initialized: now 1790564301210444 us; error 48 us; skew 500 ppm
I20260928 02:58:21.210794 13786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:21 dist-test-slave-k5rg krb5kdc[13545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:21.213685 13786 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:58:21.215000 13786 webserver.cc:460] Webserver started at http://127.12.182.66:43391/ using document root <none> and password file <none>
I20260928 02:58:21.215626 13786 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:21.215740 13786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:21.216037 13786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:21.218827 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "80d72eb8524147d28a19d7bf8ea8fa6a"
format_stamp: "Formatted at 2026-09-28 02:58:21 on dist-test-slave-k5rg"
server_key: "dcef1a87f2a43234d0b838fccc6dbca6"
server_key_iv: "3249a12843cb781f7bb443db8f20d64b"
server_key_version: "encryptionkey@0"
I20260928 02:58:21.219622 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "80d72eb8524147d28a19d7bf8ea8fa6a"
format_stamp: "Formatted at 2026-09-28 02:58:21 on dist-test-slave-k5rg"
server_key: "dcef1a87f2a43234d0b838fccc6dbca6"
server_key_iv: "3249a12843cb781f7bb443db8f20d64b"
server_key_version: "encryptionkey@0"
I20260928 02:58:21.224720 13786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 02:58:21.228286 13801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:21.229569 13786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:58:21.229769 13786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "80d72eb8524147d28a19d7bf8ea8fa6a"
format_stamp: "Formatted at 2026-09-28 02:58:21 on dist-test-slave-k5rg"
server_key: "dcef1a87f2a43234d0b838fccc6dbca6"
server_key_iv: "3249a12843cb781f7bb443db8f20d64b"
server_key_version: "encryptionkey@0"
I20260928 02:58:21.229954 13786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:21.241079 13786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:21.244539 13786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:21.244786 13786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:21.245661 13786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:21.246867 13786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:21.246950 13786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:21.247061 13786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:21.247110 13786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:21.276646 13786 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:41231
I20260928 02:58:21.276676 13913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:41231 every 8 connection(s)
I20260928 02:58:21.277869 13786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:58:21.280269 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 13786
I20260928 02:58:21.280445 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:58:21.280925 13017 external_mini_cluster.cc:1428] Setting key f6c530add88e181efa9212d6e647968c
Sep 28 02:58:21 dist-test-slave-k5rg krb5kdc[13545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
I20260928 02:58:21.297961 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.280260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:57305 (local address 127.12.182.126:45753)
0928 02:58:21.280500 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:21.280506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:21.281809 (+ 1303us) server_negotiation.cc:407] Connection header received
0928 02:58:21.283622 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:21.283628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:21.283708 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:21.283791 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:21.285303 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.286178 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.287683 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.288069 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.288175 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.291857 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:21.291865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:21.291870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:21.291959 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:21.295205 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:21.295868 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:21.295872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:21.295874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:21.296008 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:21.296400 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:21.296404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:21.296406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:21.296661 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:21.296774 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:21.297444 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 02:58:21.297648 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 02:58:21.299002 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.280753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45753 (local address 127.12.182.66:57305)
0928 02:58:21.281662 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:21.281712 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:21.283360 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:21.284054 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:21.284067 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:21.284600 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:21.285141 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:21.285158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.286323 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.286329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:21.287548 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:21.287561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.288272 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.288279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:21.288518 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.289119 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:21.289140 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:21.291620 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:21.295329 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:21.295335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:21.295348 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:21.295765 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:21.296090 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:21.296092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:21.296095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:21.296310 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:21.296752 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:21.296757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:21.297097 (+ 340us) client_negotiation.cc:769] Sending connection context
0928 02:58:21.297366 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 02:58:21.297896 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":310,"threads_started":1}
I20260928 02:58:21.300513 13914 heartbeater.cc:347] Connected to a master server at 127.12.182.126:45753
I20260928 02:58:21.300822 13914 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:21.301494 13914 heartbeater.cc:511] Master 127.12.182.126:45753 requested a full tablet report, sending...
I20260928 02:58:21.302987 13590 ts_manager.cc:194] Registered new tserver with Master: 80d72eb8524147d28a19d7bf8ea8fa6a (127.12.182.66:41231)
I20260928 02:58:21.304100 13590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:57305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:58:21.316953 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45753
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:21.614679 13919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:21.614955 13919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:21.615063 13919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:21.622161 13919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:21.622255 13919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:21.622372 13919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:58:21.631704 13919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45753
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:21.632948 13919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:21.634274 13919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:21.646783 13924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:21.646773 13925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:21.646780 13927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:21.647385 13919 server_base.cc:1034] running on GCE node
I20260928 02:58:21.648017 13919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:21.648901 13919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:21.650099 13919 hybrid_clock.cc:648] HybridClock initialized: now 1790564301650046 us; error 36 us; skew 500 ppm
I20260928 02:58:21.650405 13919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:21 dist-test-slave-k5rg krb5kdc[13545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:21.653306 13919 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:58:21.654667 13919 webserver.cc:460] Webserver started at http://127.12.182.67:38415/ using document root <none> and password file <none>
I20260928 02:58:21.655297 13919 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:21.655409 13919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:21.655694 13919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:21.658520 13919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f"
format_stamp: "Formatted at 2026-09-28 02:58:21 on dist-test-slave-k5rg"
server_key: "eb055f5b841291979b3dd46989b5c650"
server_key_iv: "95a0cf9ee02cbbf500311178765343bf"
server_key_version: "encryptionkey@0"
I20260928 02:58:21.659245 13919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f"
format_stamp: "Formatted at 2026-09-28 02:58:21 on dist-test-slave-k5rg"
server_key: "eb055f5b841291979b3dd46989b5c650"
server_key_iv: "95a0cf9ee02cbbf500311178765343bf"
server_key_version: "encryptionkey@0"
I20260928 02:58:21.664075 13919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:58:21.667303 13934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:21.668581 13919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:58:21.668745 13919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f"
format_stamp: "Formatted at 2026-09-28 02:58:21 on dist-test-slave-k5rg"
server_key: "eb055f5b841291979b3dd46989b5c650"
server_key_iv: "95a0cf9ee02cbbf500311178765343bf"
server_key_version: "encryptionkey@0"
I20260928 02:58:21.668936 13919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:21.690537 13919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:21.694334 13919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:21.694597 13919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:21.695499 13919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:21.696745 13919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:21.696825 13919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:21.696949 13919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:21.697003 13919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:21.725803 13919 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:35701
I20260928 02:58:21.725844 14046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:35701 every 8 connection(s)
I20260928 02:58:21.726985 13919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:58:21.730930 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 13919
I20260928 02:58:21.731101 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:58:21.731483 13017 external_mini_cluster.cc:1428] Setting key c12f7571ae38bbbdb117fe43a39fec7a
Sep 28 02:58:21 dist-test-slave-k5rg krb5kdc[13545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:21.745275 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.729276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:54129 (local address 127.12.182.126:45753)
0928 02:58:21.729451 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:21.729456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:21.730591 (+ 1135us) server_negotiation.cc:407] Connection header received
0928 02:58:21.732646 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:21.732651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:21.732736 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:21.732817 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:21.734492 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.735209 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.736661 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.737061 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.737168 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.740241 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:21.740246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:21.740249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:21.740279 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:21.742495 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:21.743141 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:21.743145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:21.743147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:21.743260 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:21.743652 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:21.743656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:21.743658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:21.743937 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:21.744075 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:21.744701 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 02:58:21.744943 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:58:21.746233 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.729751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45753 (local address 127.12.182.67:54129)
0928 02:58:21.730431 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:21.730472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:21.732359 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:21.733124 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:21.733133 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:21.733725 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:21.734334 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:21.734353 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.735353 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.735357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:21.736533 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:21.736545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.737259 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.737267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:21.737493 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.738122 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:21.738144 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:21.740054 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:21.742616 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:21.742624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:21.742640 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:21.743025 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:21.743357 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:21.743362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:21.743366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:21.743544 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:21.744055 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:21.744062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:21.744362 (+ 300us) client_negotiation.cc:769] Sending connection context
0928 02:58:21.744586 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 02:58:21.745057 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":247,"threads_started":1}
I20260928 02:58:21.747385 14047 heartbeater.cc:347] Connected to a master server at 127.12.182.126:45753
I20260928 02:58:21.747648 14047 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:21.748323 14047 heartbeater.cc:511] Master 127.12.182.126:45753 requested a full tablet report, sending...
I20260928 02:58:21.749521 13590 ts_manager.cc:194] Registered new tserver with Master: 3fa63b8fb92e4dd38a36f51235f3cb2f (127.12.182.67:35701)
I20260928 02:58:21.750288 13590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:54129
I20260928 02:58:21.758042 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:58:21.775004 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.763826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.12.182.126:45753)
0928 02:58:21.764018 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:21.764022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:21.764478 (+ 456us) server_negotiation.cc:407] Connection header received
0928 02:58:21.764630 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:21.764633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:21.764691 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:21.764765 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:21.766122 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.766828 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.767993 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.768379 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.768471 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.770554 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:21.770559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:21.770564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:21.770597 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:21.772887 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:21.773495 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:21.773499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:21.773501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:21.773671 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:21.773932 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:21.773935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:21.773937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:21.774231 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:21.774402 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:21.774622 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:58:21.774775 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:58:21.780790 13590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:58:21.787407 13590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:21.820377 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.798423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:41231 (local address 127.0.0.1:34592)
0928 02:58:21.805126 (+ 6703us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:21.805161 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:21.805324 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:21.813195 (+ 7871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:21.813203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:21.813250 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:21.813785 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:21.813794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.815552 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.815556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:21.817735 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:21.817748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.819313 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.819319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:21.819513 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.819580 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 02:58:21.819709 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 02:58:21.819895 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6456,"mutex_wait_us":233,"thread_start_us":6323,"threads_started":1}
I20260928 02:58:21.820515 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.798661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.12.182.66:41231)
0928 02:58:21.812604 (+ 13943us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:21.812613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:21.812648 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:58:21.812743 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:21.812750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:21.812979 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:21.813556 (+ 577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:21.813926 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.815426 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.817875 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.819098 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.819236 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.819457 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:21.820024 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 02:58:21.820251 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13796,"thread_start_us":13682,"threads_started":1}
I20260928 02:58:21.821108 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.799047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:35701 (local address 127.0.0.1:36696)
0928 02:58:21.809419 (+ 10372us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:21.809439 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:21.809563 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:21.813027 (+ 3464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:21.813035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:21.813063 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:21.813505 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:21.813517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.815537 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.815542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:21.818255 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:21.818270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.820230 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.820237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:21.820436 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.820467 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:58:21.820545 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:58:21.820662 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10243,"mutex_wait_us":172,"spinlock_wait_cycles":68352,"thread_start_us":10575,"threads_started":1}
I20260928 02:58:21.821159 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.809959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43387 (local address 127.0.0.1:45278)
0928 02:58:21.812010 (+ 2051us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:21.812028 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:21.812134 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:21.813432 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:21.813436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:21.813467 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:21.814120 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:21.814128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.815477 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.815481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:21.818265 (+ 2784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:21.818275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.819730 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.819783 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:21.819960 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.819991 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:58:21.820073 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:58:21.820211 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1900,"mutex_wait_us":30,"thread_start_us":1757,"threads_started":1}
I20260928 02:58:21.821094 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.805834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.12.182.65:43387)
0928 02:58:21.812907 (+ 7073us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:21.812915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:21.812938 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:58:21.813032 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:21.813039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:21.813241 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:21.813432 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:21.814255 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.815304 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.818456 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.819507 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.819644 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.819873 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:21.820496 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 02:58:21.820667 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6924,"thread_start_us":6815,"threads_started":1}
I20260928 02:58:21.822177 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:21.799231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.12.182.67:35701)
0928 02:58:21.812379 (+ 13148us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:21.812390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:21.812428 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:58:21.812534 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:21.812544 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:21.812817 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:21.813001 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:21.813717 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.815373 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.818424 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:21.819751 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:21.819872 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:21.820091 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:21.821692 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0928 02:58:21.821851 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12952,"thread_start_us":12759,"threads_started":1}
I20260928 02:58:21.824826 13849 tablet_service.cc:1467] Processing CreateTablet for tablet 62ddacab957545e1825da36de3661533 (DEFAULT_TABLE table=test-table [id=07a0935b341848cca497f9a16a8fcd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:21.825692 13715 tablet_service.cc:1467] Processing CreateTablet for tablet 62ddacab957545e1825da36de3661533 (DEFAULT_TABLE table=test-table [id=07a0935b341848cca497f9a16a8fcd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:21.826367 13849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ddacab957545e1825da36de3661533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:21.827078 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ddacab957545e1825da36de3661533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:21.827607 13982 tablet_service.cc:1467] Processing CreateTablet for tablet 62ddacab957545e1825da36de3661533 (DEFAULT_TABLE table=test-table [id=07a0935b341848cca497f9a16a8fcd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:21.828749 13982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ddacab957545e1825da36de3661533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:21.844465 14067 tablet_bootstrap.cc:492] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516: Bootstrap starting.
I20260928 02:58:21.846390 14065 tablet_bootstrap.cc:492] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a: Bootstrap starting.
I20260928 02:58:21.846565 14066 tablet_bootstrap.cc:492] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f: Bootstrap starting.
I20260928 02:58:21.848155 14067 tablet_bootstrap.cc:654] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:21.849315 14067 log.cc:824] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:21.849905 14066 tablet_bootstrap.cc:654] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:21.850586 14065 tablet_bootstrap.cc:654] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:21.851102 14066 log.cc:824] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:21.851735 14065 log.cc:824] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:21.851959 14067 tablet_bootstrap.cc:492] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516: No bootstrap required, opened a new log
I20260928 02:58:21.852236 14067 ts_tablet_manager.cc:1397] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260928 02:58:21.854066 14066 tablet_bootstrap.cc:492] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f: No bootstrap required, opened a new log
I20260928 02:58:21.854300 14066 ts_tablet_manager.cc:1397] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260928 02:58:21.855962 14065 tablet_bootstrap.cc:492] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a: No bootstrap required, opened a new log
I20260928 02:58:21.856206 14065 ts_tablet_manager.cc:1397] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260928 02:58:21.859347 14067 raft_consensus.cc:348] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41231 } } peers { permanent_uuid: "f5c137b2aab54fd78940802511bd7516" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43387 } } peers { permanent_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 35701 } }
I20260928 02:58:21.859716 14067 raft_consensus.cc:374] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:21.859808 14067 raft_consensus.cc:729] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5c137b2aab54fd78940802511bd7516, State: Initialized, Role: FOLLOWER
I20260928 02:58:21.860479 14067 consensus_queue.cc:260] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41231 } } peers { permanent_uuid: "f5c137b2aab54fd78940802511bd7516" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43387 } } peers { permanent_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 35701 } }
I20260928 02:58:21.863910 13780 heartbeater.cc:503] Master 127.12.182.126:45753 was elected leader, sending a full tablet report...
I20260928 02:58:21.864336 14066 raft_consensus.cc:348] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41231 } } peers { permanent_uuid: "f5c137b2aab54fd78940802511bd7516" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43387 } } peers { permanent_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 35701 } }
I20260928 02:58:21.864728 14066 raft_consensus.cc:374] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:21.864812 14066 raft_consensus.cc:729] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa63b8fb92e4dd38a36f51235f3cb2f, State: Initialized, Role: FOLLOWER
I20260928 02:58:21.865494 14066 consensus_queue.cc:260] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41231 } } peers { permanent_uuid: "f5c137b2aab54fd78940802511bd7516" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43387 } } peers { permanent_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 35701 } }
I20260928 02:58:21.866078 14065 raft_consensus.cc:348] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41231 } } peers { permanent_uuid: "f5c137b2aab54fd78940802511bd7516" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43387 } } peers { permanent_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 35701 } }
I20260928 02:58:21.866590 14065 raft_consensus.cc:374] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:21.866719 14065 raft_consensus.cc:729] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d72eb8524147d28a19d7bf8ea8fa6a, State: Initialized, Role: FOLLOWER
I20260928 02:58:21.867462 14065 consensus_queue.cc:260] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41231 } } peers { permanent_uuid: "f5c137b2aab54fd78940802511bd7516" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43387 } } peers { permanent_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 35701 } }
I20260928 02:58:21.868598 14067 ts_tablet_manager.cc:1428] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260928 02:58:21.869571 14047 heartbeater.cc:503] Master 127.12.182.126:45753 was elected leader, sending a full tablet report...
I20260928 02:58:21.870237 14066 ts_tablet_manager.cc:1428] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260928 02:58:21.871094 13914 heartbeater.cc:503] Master 127.12.182.126:45753 was elected leader, sending a full tablet report...
I20260928 02:58:21.871651 14065 ts_tablet_manager.cc:1428] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
W20260928 02:58:21.978454 14048 tablet.cc:2369] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:58:22.029779 13915 tablet.cc:2369] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:58:22.085217 13781 tablet.cc:2369] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:58:22.108958 14072 raft_consensus.cc:482] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:22.109257 14072 raft_consensus.cc:504] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41231 } } peers { permanent_uuid: "f5c137b2aab54fd78940802511bd7516" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43387 } } peers { permanent_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 35701 } }
I20260928 02:58:22.110877 14072 leader_election.cc:288] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80d72eb8524147d28a19d7bf8ea8fa6a (127.12.182.66:41231), f5c137b2aab54fd78940802511bd7516 (127.12.182.65:43387)
I20260928 02:58:22.117460 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.111384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:59185 (local address 127.12.182.66:41231)
0928 02:58:22.111692 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:22.111697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:22.111717 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:58:22.111794 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:22.111798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:22.111874 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:22.111968 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:22.112701 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.113850 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.115902 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.116658 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.116790 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:22.116861 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:22.117103 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 02:58:22.117225 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 02:58:22.117460 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.111142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:41231 (local address 127.12.182.67:59185)
0928 02:58:22.111331 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:22.111350 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:22.111468 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:22.112144 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:22.112150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:22.112177 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:22.112430 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:22.112444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.114028 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.114033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:22.115761 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:22.115773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.116795 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.116800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:22.116968 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:22.116992 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:58:22.117071 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:58:22.117194 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":18}
I20260928 02:58:22.118372 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ddacab957545e1825da36de3661533" candidate_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" is_pre_election: true
I20260928 02:58:22.118597 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.111997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:54815 (local address 127.12.182.65:43387)
0928 02:58:22.112197 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:22.112203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:22.112869 (+ 666us) server_negotiation.cc:407] Connection header received
0928 02:58:22.113179 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:22.113183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:22.113269 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:22.113350 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:22.114106 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.115126 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.116778 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.117570 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.117899 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:22.117969 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:22.118227 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:58:22.118354 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:58:22.118666 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.111540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43387 (local address 127.12.182.67:54815)
0928 02:58:22.112777 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:22.112794 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:22.112984 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:22.113450 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:22.113455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:22.113516 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:22.113935 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:22.113948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.115280 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.115283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:22.116625 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:22.116637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.117922 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.117928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:22.118085 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:22.118107 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:58:22.118183 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:58:22.118299 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"mutex_wait_us":23,"thread_start_us":912,"threads_started":1}
I20260928 02:58:22.118878 13869 raft_consensus.cc:2454] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fa63b8fb92e4dd38a36f51235f3cb2f in term 0.
I20260928 02:58:22.119493 13935 leader_election.cc:302] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fa63b8fb92e4dd38a36f51235f3cb2f, 80d72eb8524147d28a19d7bf8ea8fa6a; no voters:
I20260928 02:58:22.119540 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ddacab957545e1825da36de3661533" candidate_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c137b2aab54fd78940802511bd7516" is_pre_election: true
I20260928 02:58:22.119834 14072 raft_consensus.cc:2781] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:22.119971 14072 raft_consensus.cc:482] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:22.120078 14072 raft_consensus.cc:3037] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:22.120059 13735 raft_consensus.cc:2454] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fa63b8fb92e4dd38a36f51235f3cb2f in term 0.
I20260928 02:58:22.122376 14072 raft_consensus.cc:504] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41231 } } peers { permanent_uuid: "f5c137b2aab54fd78940802511bd7516" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43387 } } peers { permanent_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 35701 } }
I20260928 02:58:22.122921 14072 leader_election.cc:288] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [CANDIDATE]: Term 1 election: Requested vote from peers 80d72eb8524147d28a19d7bf8ea8fa6a (127.12.182.66:41231), f5c137b2aab54fd78940802511bd7516 (127.12.182.65:43387)
I20260928 02:58:22.123304 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ddacab957545e1825da36de3661533" candidate_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a"
I20260928 02:58:22.123351 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ddacab957545e1825da36de3661533" candidate_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c137b2aab54fd78940802511bd7516"
I20260928 02:58:22.123510 13735 raft_consensus.cc:3037] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:22.123510 13869 raft_consensus.cc:3037] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:22.126541 13735 raft_consensus.cc:2454] T 62ddacab957545e1825da36de3661533 P f5c137b2aab54fd78940802511bd7516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fa63b8fb92e4dd38a36f51235f3cb2f in term 1.
I20260928 02:58:22.126550 13869 raft_consensus.cc:2454] T 62ddacab957545e1825da36de3661533 P 80d72eb8524147d28a19d7bf8ea8fa6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fa63b8fb92e4dd38a36f51235f3cb2f in term 1.
I20260928 02:58:22.126891 13935 leader_election.cc:302] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fa63b8fb92e4dd38a36f51235f3cb2f, f5c137b2aab54fd78940802511bd7516; no voters:
I20260928 02:58:22.127126 14072 raft_consensus.cc:2781] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:22.127516 14072 raft_consensus.cc:686] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [term 1 LEADER]: Becoming Leader. State: Replica: 3fa63b8fb92e4dd38a36f51235f3cb2f, State: Running, Role: LEADER
I20260928 02:58:22.127942 14072 consensus_queue.cc:237] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41231 } } peers { permanent_uuid: "f5c137b2aab54fd78940802511bd7516" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43387 } } peers { permanent_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 35701 } }
I20260928 02:58:22.132022 13589 catalog_manager.cc:5495] T 62ddacab957545e1825da36de3661533 P 3fa63b8fb92e4dd38a36f51235f3cb2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fa63b8fb92e4dd38a36f51235f3cb2f (127.12.182.67). New cstate: current_term: 1 leader_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d72eb8524147d28a19d7bf8ea8fa6a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5c137b2aab54fd78940802511bd7516" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fa63b8fb92e4dd38a36f51235f3cb2f" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 35701 } health_report { overall_health: HEALTHY } } }
Sep 28 02:58:22 dist-test-slave-k5rg krb5kdc[13545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:58:22.177945 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.164073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.12.182.65:43387)
0928 02:58:22.164245 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:22.164252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:22.164686 (+ 434us) server_negotiation.cc:407] Connection header received
0928 02:58:22.164848 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:22.164852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:22.164918 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:22.164999 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:22.166613 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.167372 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.168616 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.169055 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.169165 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:22.172836 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:22.172850 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:22.172865 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:22.172903 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:22.175595 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.176217 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.176223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.176225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.176352 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.176680 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.176683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.176685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.177040 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:22.177244 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:22.177456 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:58:22.177644 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 02:58:22 dist-test-slave-k5rg krb5kdc[13545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:58:22 dist-test-slave-k5rg krb5kdc[13545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:58:22.218356 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.205579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.12.182.65:43387)
0928 02:58:22.205789 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:22.205796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:22.206111 (+ 315us) server_negotiation.cc:407] Connection header received
0928 02:58:22.206292 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:22.206297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:22.206362 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:22.206446 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:22.207747 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.208506 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.209777 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.210169 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.210274 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:22.213751 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:22.213756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:22.213759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:22.213792 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:22.216015 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.216581 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.216585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.216588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.216706 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.217067 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.217070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.217072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.217326 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:22.217588 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:22.217862 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:58:22.218071 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 02:58:22 dist-test-slave-k5rg krb5kdc[13545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:58:22 dist-test-slave-k5rg krb5kdc[13545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:58:22.257481 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.244835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45312 (local address 127.12.182.65:43387)
0928 02:58:22.245066 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:22.245072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:22.245255 (+ 183us) server_negotiation.cc:407] Connection header received
0928 02:58:22.245478 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:22.245483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:22.245553 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:22.245705 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:22.246990 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.247664 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.248887 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.249292 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.249375 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:22.252829 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:22.252833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:22.252837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:22.252870 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:22.255152 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.255776 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.255781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.255783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.255907 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.256255 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.256259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.256261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.256574 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:22.256755 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:22.256985 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:58:22.257151 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260928 02:58:22.259157 13695 tablet_service.cc:2908] Rejecting scan request for tablet 62ddacab957545e1825da36de3661533: Uninitialized: clean time has not yet been initialized
I20260928 02:58:22.261806 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 13652
I20260928 02:58:22.274097 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.262663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.12.182.65:43387)
0928 02:58:22.262818 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:22.262824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:22.263580 (+ 756us) server_negotiation.cc:407] Connection header received
0928 02:58:22.263724 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:22.263729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:22.263799 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:22.263888 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:22.264945 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.265642 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.266789 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.267180 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.267293 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:22.269294 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:22.269301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:22.269305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:22.269333 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:22.271645 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.272333 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.272338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.272342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.272463 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.272787 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.272792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.272796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.273072 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:22.273365 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:22.273578 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 02:58:22.273797 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:58:22.274691 13775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:22.366469 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 13652
I20260928 02:58:22.380218 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 13786
Sep 28 02:58:22 dist-test-slave-k5rg krb5kdc[13545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 02:58:22.393762 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.380970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.12.182.66:41231)
0928 02:58:22.381155 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:22.381161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:22.381181 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:58:22.381244 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:22.381249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:22.381309 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:22.381389 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:22.382554 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.383271 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.384514 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.384909 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.385019 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:22.388760 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:22.388774 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:22.388789 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:22.388823 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:22.391444 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.392016 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.392024 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.392026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.392159 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.392465 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.392468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.392470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.392833 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:22.393112 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:22.393272 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:58:22.393437 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:58:22.394394 13909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:22.478756 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 13786
I20260928 02:58:22.491103 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 13919
Sep 28 02:58:22 dist-test-slave-k5rg krb5kdc[13545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 02:58:22.504866 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.491765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.12.182.67:35701)
0928 02:58:22.492216 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:22.492220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:22.492237 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:58:22.492291 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:22.492295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:22.492360 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:22.492446 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:22.493615 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.494301 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.495558 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:22.495949 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:22.496046 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:22.499871 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:22.499884 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:22.499901 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:22.499933 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:22.502512 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.503183 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.503189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.503191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.503328 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:22.503635 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:22.503637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:22.503639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:22.503955 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:22.504160 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:22.504359 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 02:58:22.504508 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
I20260928 02:58:22.505487 14042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:22.592303 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.591557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:41231 (local address 127.12.182.67:48191)
0928 02:58:22.591966 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:22.592110 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.66:41231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253}
W20260928 02:58:22.592672 13935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.182.66:41231: connect: Connection refused (error 111)
I20260928 02:58:22.592815 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:22.592357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43387 (local address 127.12.182.67:54629)
0928 02:58:22.592530 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:22.592618 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:58:22.593343 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 13919
I20260928 02:58:22.606269 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 13558
I20260928 02:58:22.607347 13620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:22.712280 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 13558
2026-09-28T02:58:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4045 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:22 dist-test-slave-k5rg krb5kdc[14096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:22 dist-test-slave-k5rg krb5kdc[14096](info): set up 2 sockets
Sep 28 02:58:22 dist-test-slave-k5rg krb5kdc[14096](info): commencing operation
krb5kdc: starting...
W20260928 02:58:23.697796 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:58:23 dist-test-slave-k5rg krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:58:23.854403 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33885
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:24.152678 14109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:24.152951 14109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:24.153065 14109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:24.160049 14109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:24.160143 14109 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:58:24.160207 14109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:24.160285 14109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:24.160346 14109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:24.160425 14109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:24.170001 14109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33885
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:24.171157 14109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:24.172448 14109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:24.183342 14114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:24.183394 14117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:24.183457 14115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:24.184151 14109 server_base.cc:1034] running on GCE node
I20260928 02:58:24.184868 14109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:24.186226 14109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:24.187427 14109 hybrid_clock.cc:648] HybridClock initialized: now 1790564304187374 us; error 36 us; skew 500 ppm
I20260928 02:58:24.187723 14109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:24 dist-test-slave-k5rg krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564304, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:24.190510 14109 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:58:24.191833 14109 webserver.cc:460] Webserver started at http://127.12.182.126:35191/ using document root <none> and password file <none>
I20260928 02:58:24.192456 14109 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:24.192559 14109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:24.192845 14109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:24.195708 14109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "0134da097ddd44f6bc657fbe83068d87"
format_stamp: "Formatted at 2026-09-28 02:58:24 on dist-test-slave-k5rg"
server_key: "4a977917180a5b8f9d170cac81dd8e9e"
server_key_iv: "a032c566dcefd47b11c33b2234482ef6"
server_key_version: "encryptionkey@0"
I20260928 02:58:24.196400 14109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "0134da097ddd44f6bc657fbe83068d87"
format_stamp: "Formatted at 2026-09-28 02:58:24 on dist-test-slave-k5rg"
server_key: "4a977917180a5b8f9d170cac81dd8e9e"
server_key_iv: "a032c566dcefd47b11c33b2234482ef6"
server_key_version: "encryptionkey@0"
I20260928 02:58:24.201074 14109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 02:58:24.204332 14124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:24.205518 14109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:58:24.205690 14109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "0134da097ddd44f6bc657fbe83068d87"
format_stamp: "Formatted at 2026-09-28 02:58:24 on dist-test-slave-k5rg"
server_key: "4a977917180a5b8f9d170cac81dd8e9e"
server_key_iv: "a032c566dcefd47b11c33b2234482ef6"
server_key_version: "encryptionkey@0"
I20260928 02:58:24.205873 14109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:24.251021 14109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:24.254724 14109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:24.254951 14109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:24.273973 14109 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:33885
I20260928 02:58:24.274008 14185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:33885 every 8 connection(s)
I20260928 02:58:24.275218 14109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:24.277030 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 14109
I20260928 02:58:24.277257 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:24.278061 13017 external_mini_cluster.cc:1428] Setting key 60bd533d322071a5b73d2686abf7a4b4
I20260928 02:58:24.279445 14186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:58:24 dist-test-slave-k5rg krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:24.292486 14186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87: Bootstrap starting.
I20260928 02:58:24.296298 14186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:24.297451 14186 log.cc:824] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:24.300496 14186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87: No bootstrap required, opened a new log
I20260928 02:58:24.300693 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:24.279988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.12.182.126:33885)
0928 02:58:24.280714 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:24.280730 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:24.280784 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:58:24.282704 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:24.282730 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:24.283214 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:24.283631 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:24.284930 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:24.286938 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:24.288086 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:24.289063 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:24.289254 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:24.293027 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:24.293037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:24.293055 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:24.293095 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:24.296653 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:24.297250 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:24.297262 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:24.297270 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:24.297475 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:24.297826 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:24.297831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:24.297834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:24.298422 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:24.298640 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:24.299064 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 02:58:24.299471 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":314,"threads_started":1}
I20260928 02:58:24.308017 14186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134da097ddd44f6bc657fbe83068d87" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33885 } }
I20260928 02:58:24.308408 14186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:24.308519 14186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0134da097ddd44f6bc657fbe83068d87, State: Initialized, Role: FOLLOWER
I20260928 02:58:24.309101 14186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134da097ddd44f6bc657fbe83068d87" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33885 } }
I20260928 02:58:24.309317 14186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:24.309440 14186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:24.309590 14186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:24.311698 14186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134da097ddd44f6bc657fbe83068d87" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33885 } }
I20260928 02:58:24.312211 14186 leader_election.cc:302] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0134da097ddd44f6bc657fbe83068d87; no voters:
I20260928 02:58:24.312825 14186 leader_election.cc:288] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:24.312920 14191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:24.313555 14191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [term 1 LEADER]: Becoming Leader. State: Replica: 0134da097ddd44f6bc657fbe83068d87, State: Running, Role: LEADER
I20260928 02:58:24.314136 14191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134da097ddd44f6bc657fbe83068d87" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33885 } }
I20260928 02:58:24.314916 14186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:24.316646 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0134da097ddd44f6bc657fbe83068d87. Latest consensus state: current_term: 1 leader_uuid: "0134da097ddd44f6bc657fbe83068d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134da097ddd44f6bc657fbe83068d87" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33885 } } }
I20260928 02:58:24.317009 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:24.317574 14192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0134da097ddd44f6bc657fbe83068d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134da097ddd44f6bc657fbe83068d87" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33885 } } }
I20260928 02:58:24.317873 14192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:24.318678 14197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:24.323760 14197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:24.331713 14197 catalog_manager.cc:1347] Generated new cluster ID: b57f3839404d439b87a056542e0057c3
I20260928 02:58:24.331815 14197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:24.357358 14197 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:24.358685 14197 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:24.366946 14197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87: Generated new TSK 0
I20260928 02:58:24.367730 14197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:58:24.410964 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33885
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:24.709080 14212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:24.709332 14212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:24.709463 14212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:24.716496 14212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:24.716588 14212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:24.716702 14212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:58:24.716804 14212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:58:24.726207 14212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.182.126:33885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:24.727332 14212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:24.728547 14212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:24.740338 14217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:24.740311 14218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:24.740389 14220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:24.740947 14212 server_base.cc:1034] running on GCE node
I20260928 02:58:24.741530 14212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:24.742401 14212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:24.743643 14212 hybrid_clock.cc:648] HybridClock initialized: now 1790564304743570 us; error 43 us; skew 500 ppm
I20260928 02:58:24.743955 14212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:24 dist-test-slave-k5rg krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564304, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:24.746941 14212 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:58:24.748226 14212 webserver.cc:460] Webserver started at http://127.12.182.65:36851/ using document root <none> and password file <none>
I20260928 02:58:24.748883 14212 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:24.748977 14212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:24.749260 14212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:24.751976 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "45ce12d4b8224c93917ac604762f9b9e"
format_stamp: "Formatted at 2026-09-28 02:58:24 on dist-test-slave-k5rg"
server_key: "3994fa2deb43ddac32db56111de0d8ee"
server_key_iv: "5f7b0eccc4206891dbab1440e669cd73"
server_key_version: "encryptionkey@0"
I20260928 02:58:24.752710 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "45ce12d4b8224c93917ac604762f9b9e"
format_stamp: "Formatted at 2026-09-28 02:58:24 on dist-test-slave-k5rg"
server_key: "3994fa2deb43ddac32db56111de0d8ee"
server_key_iv: "5f7b0eccc4206891dbab1440e669cd73"
server_key_version: "encryptionkey@0"
I20260928 02:58:24.757511 14212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 02:58:24.760768 14227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:24.762075 14212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:58:24.762226 14212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "45ce12d4b8224c93917ac604762f9b9e"
format_stamp: "Formatted at 2026-09-28 02:58:24 on dist-test-slave-k5rg"
server_key: "3994fa2deb43ddac32db56111de0d8ee"
server_key_iv: "5f7b0eccc4206891dbab1440e669cd73"
server_key_version: "encryptionkey@0"
I20260928 02:58:24.762428 14212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:24.778894 14212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:24.782634 14212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:24.782857 14212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:24.785501 14212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:24.785596 14212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:24.785771 14212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:24.785838 14212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:58:24 dist-test-slave-k5rg krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564304, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:24.811508 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:24.785944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:48231 (local address 127.12.182.126:33885)
0928 02:58:24.786112 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:24.786118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:24.790412 (+ 4294us) server_negotiation.cc:407] Connection header received
0928 02:58:24.793636 (+ 3224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:24.793644 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:24.793743 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:24.793856 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:24.795952 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:24.796889 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:24.799157 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:24.799806 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:24.799937 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:24.805028 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:24.805035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:24.805039 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:24.805085 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:24.807929 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:24.808764 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:24.808768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:24.808770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:24.808884 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:24.809409 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:24.809414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:24.809418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:24.809769 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:24.809907 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:24.810955 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0928 02:58:24.811182 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:58:24.812906 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:24.786542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33885 (local address 127.12.182.65:48231)
0928 02:58:24.790158 (+ 3616us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:24.790246 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:24.793268 (+ 3022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:24.794155 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:24.794167 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:24.794852 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:24.795751 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:24.795789 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:24.797076 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:24.797084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:24.799006 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:24.799024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:24.800084 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:24.800093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:24.800414 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:24.801271 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:24.801310 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:24.804736 (+ 3426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:24.808061 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:24.808070 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:24.808090 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:24.808613 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:24.808988 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:24.808993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:24.808996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:24.809252 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:24.809986 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:24.809998 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:24.810561 (+ 563us) client_negotiation.cc:769] Sending connection context
0928 02:58:24.810863 (+ 302us) client_negotiation.cc:240] Negotiation successful
0928 02:58:24.811415 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3244,"thread_start_us":2965,"threads_started":1}
I20260928 02:58:24.824687 14212 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:43529
I20260928 02:58:24.824723 14346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:43529 every 8 connection(s)
I20260928 02:58:24.826006 14212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:58:24.828017 14347 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33885
I20260928 02:58:24.828249 14347 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:24.828975 14347 heartbeater.cc:511] Master 127.12.182.126:33885 requested a full tablet report, sending...
I20260928 02:58:24.831148 14141 ts_manager.cc:194] Registered new tserver with Master: 45ce12d4b8224c93917ac604762f9b9e (127.12.182.65:43529)
I20260928 02:58:24.832451 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:24.819657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:39605 (local address 127.12.182.126:33885)
0928 02:58:24.819800 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:24.819804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:24.820442 (+ 638us) server_negotiation.cc:407] Connection header received
0928 02:58:24.820597 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:24.820603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:24.820671 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:24.820756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:24.822127 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:24.823110 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:24.824744 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:24.825184 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:24.825288 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:24.827211 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:24.827217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:24.827221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:24.827257 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:24.830105 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:24.830783 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:24.830786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:24.830788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:24.830935 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:24.831285 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:24.831290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:24.831293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:24.831562 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:24.831666 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:24.832060 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 02:58:24.832207 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:58:24.832762 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:24.819810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33885 (local address 127.12.182.65:39605)
0928 02:58:24.820351 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:24.820365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:24.820492 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:24.820933 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:24.820938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:24.821493 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:24.821943 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:24.821957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:24.823233 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:24.823239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:24.824623 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:24.824638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:24.825353 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:24.825359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:24.825538 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:24.826154 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:24.826173 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:24.827009 (+ 836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:24.830218 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:24.830222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:24.830226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:24.830684 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:24.831033 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:24.831037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:24.831041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:24.831175 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:24.831672 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:24.831676 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:24.831848 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 02:58:24.831994 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 02:58:24.832264 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":299,"threads_started":1}
I20260928 02:58:24.832990 14141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:48231
I20260928 02:58:24.836042 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 14212
I20260928 02:58:24.836274 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:58:24.836925 13017 external_mini_cluster.cc:1428] Setting key 13bed007c169f78618f17c3b37caf2c4
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:58:24.870316 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33885
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:25.169239 14351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:25.169507 14351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:25.169678 14351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:25.176800 14351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:25.176909 14351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:25.177040 14351 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:58:25.177150 14351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:58:25.186601 14351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.182.126:33885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:25.187789 14351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:25.189043 14351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:25.201318 14359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:25.201342 14357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:25.201555 14356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:25.202204 14351 server_base.cc:1034] running on GCE node
I20260928 02:58:25.202732 14351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:25.203540 14351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:25.204744 14351 hybrid_clock.cc:648] HybridClock initialized: now 1790564305204688 us; error 33 us; skew 500 ppm
I20260928 02:58:25.204993 14351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:25 dist-test-slave-k5rg krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:25.210836 14351 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:58:25.212214 14351 webserver.cc:460] Webserver started at http://127.12.182.66:39131/ using document root <none> and password file <none>
I20260928 02:58:25.212843 14351 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:25.212932 14351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:25.213177 14351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:25.216107 14351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "7bacb67f7e544762a9213e1aa0a7e422"
format_stamp: "Formatted at 2026-09-28 02:58:25 on dist-test-slave-k5rg"
server_key: "d50d8ecbbe1a0d3b04b22f2fa0c46283"
server_key_iv: "48570dea9360148934e814a47d1a85ec"
server_key_version: "encryptionkey@0"
I20260928 02:58:25.216805 14351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "7bacb67f7e544762a9213e1aa0a7e422"
format_stamp: "Formatted at 2026-09-28 02:58:25 on dist-test-slave-k5rg"
server_key: "d50d8ecbbe1a0d3b04b22f2fa0c46283"
server_key_iv: "48570dea9360148934e814a47d1a85ec"
server_key_version: "encryptionkey@0"
I20260928 02:58:25.221449 14351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260928 02:58:25.224711 14366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:25.226002 14351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:58:25.226122 14351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "7bacb67f7e544762a9213e1aa0a7e422"
format_stamp: "Formatted at 2026-09-28 02:58:25 on dist-test-slave-k5rg"
server_key: "d50d8ecbbe1a0d3b04b22f2fa0c46283"
server_key_iv: "48570dea9360148934e814a47d1a85ec"
server_key_version: "encryptionkey@0"
I20260928 02:58:25.226289 14351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:25.243170 14351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:25.246712 14351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:25.246920 14351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:25.249308 14351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:25.249382 14351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:25.249496 14351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:25.249545 14351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:58:25 dist-test-slave-k5rg krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:25.271773 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.249839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:41813 (local address 127.12.182.126:33885)
0928 02:58:25.250027 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:25.250033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:25.251341 (+ 1308us) server_negotiation.cc:407] Connection header received
0928 02:58:25.253997 (+ 2656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:25.254001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:25.254067 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:25.254159 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:25.256374 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.257308 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.260000 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.260560 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.260675 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.264935 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:25.264939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:25.264943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:25.264976 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:25.268041 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:25.268969 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:25.268974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:25.268978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:25.269147 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:25.269693 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:25.269698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:25.269701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:25.270064 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:25.270203 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:25.271121 (+ 918us) server_negotiation.cc:299] Negotiation successful
0928 02:58:25.271378 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:58:25.272533 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.250361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33885 (local address 127.12.182.66:41813)
0928 02:58:25.251155 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:25.251213 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:25.253705 (+ 2492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:25.254446 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:25.254458 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:25.255262 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:25.256183 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.256210 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.257458 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.257464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.259815 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.259836 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.260798 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.260806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.261077 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.261802 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:25.261838 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:25.264741 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:25.268191 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:25.268200 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:25.268222 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:25.268819 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:25.269257 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:25.269263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:25.269267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:25.269552 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:25.270177 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:25.270188 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:25.270661 (+ 473us) client_negotiation.cc:769] Sending connection context
0928 02:58:25.270986 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 02:58:25.271382 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":311,"threads_started":1}
I20260928 02:58:25.301352 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.288151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:43017 (local address 127.12.182.126:33885)
0928 02:58:25.288353 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:25.288360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:25.288792 (+ 432us) server_negotiation.cc:407] Connection header received
0928 02:58:25.288923 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:25.288929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:25.289020 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:25.289135 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:25.290066 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.290773 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.292496 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.293058 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.293178 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.295126 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:25.295133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:25.295139 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:25.295179 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:25.298364 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:25.299123 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:25.299129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:25.299132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:25.299286 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:25.299779 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:25.299784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:25.299788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:25.300196 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:25.300354 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:25.300784 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 02:58:25.300995 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:58:25.301402 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.288098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33885 (local address 127.12.182.66:43017)
0928 02:58:25.288712 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:25.288727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:25.288833 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:25.289143 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:25.289146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:25.289557 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:25.289919 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.289930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.290903 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.290912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.292377 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.292390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.293421 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.293425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.293573 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.294239 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:25.294259 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:25.294953 (+ 694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:25.298479 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:25.298487 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:25.298491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:25.298987 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:25.299429 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:25.299437 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:25.299441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:25.299620 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:25.300335 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:25.300340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:25.300523 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 02:58:25.300663 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:58:25.300965 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":315,"threads_started":1}
I20260928 02:58:25.306569 14351 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:43591
I20260928 02:58:25.306591 14485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:43591 every 8 connection(s)
I20260928 02:58:25.307672 14351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:58:25.309513 14486 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33885
I20260928 02:58:25.309739 14486 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:25.310315 14486 heartbeater.cc:511] Master 127.12.182.126:33885 requested a full tablet report, sending...
I20260928 02:58:25.311506 14140 ts_manager.cc:194] Registered new tserver with Master: 7bacb67f7e544762a9213e1aa0a7e422 (127.12.182.66:43591)
I20260928 02:58:25.312340 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:41813
I20260928 02:58:25.313256 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 14351
I20260928 02:58:25.313462 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:58:25.313915 13017 external_mini_cluster.cc:1428] Setting key ff27a4e1943027112e9805058aee48a9
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:58:25.346935 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33885
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:25.648518 14490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:25.648763 14490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:25.648896 14490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:25.655807 14490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:25.655908 14490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:25.656038 14490 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:58:25.656136 14490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:58:25.665285 14490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.182.126:33885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:25.666399 14490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:25.667665 14490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:25.679425 14495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:25.679392 14496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:25.679468 14498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:25.680119 14490 server_base.cc:1034] running on GCE node
I20260928 02:58:25.680675 14490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:25.681485 14490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:25.682685 14490 hybrid_clock.cc:648] HybridClock initialized: now 1790564305682632 us; error 49 us; skew 500 ppm
I20260928 02:58:25.682945 14490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:25 dist-test-slave-k5rg krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:25.685814 14490 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:58:25.687139 14490 webserver.cc:460] Webserver started at http://127.12.182.67:34051/ using document root <none> and password file <none>
I20260928 02:58:25.687736 14490 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:25.687824 14490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:25.688117 14490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:25.690814 14490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "52ac678b40f84a0eab1100cb5bf32e1d"
format_stamp: "Formatted at 2026-09-28 02:58:25 on dist-test-slave-k5rg"
server_key: "1f313eec9ec4d26f3f5f56a4597f90fe"
server_key_iv: "dd9301b5dca135f902d002c1b07a2dca"
server_key_version: "encryptionkey@0"
I20260928 02:58:25.691536 14490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "52ac678b40f84a0eab1100cb5bf32e1d"
format_stamp: "Formatted at 2026-09-28 02:58:25 on dist-test-slave-k5rg"
server_key: "1f313eec9ec4d26f3f5f56a4597f90fe"
server_key_iv: "dd9301b5dca135f902d002c1b07a2dca"
server_key_version: "encryptionkey@0"
I20260928 02:58:25.696096 14490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:58:25.699169 14505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:25.700335 14490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:58:25.700462 14490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "52ac678b40f84a0eab1100cb5bf32e1d"
format_stamp: "Formatted at 2026-09-28 02:58:25 on dist-test-slave-k5rg"
server_key: "1f313eec9ec4d26f3f5f56a4597f90fe"
server_key_iv: "dd9301b5dca135f902d002c1b07a2dca"
server_key_version: "encryptionkey@0"
I20260928 02:58:25.700637 14490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:25.734071 14490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:25.738075 14490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:25.738304 14490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:25.740767 14490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:25.740846 14490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:25.740983 14490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:25.741037 14490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:58:25 dist-test-slave-k5rg krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:25.763527 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.741266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:56473 (local address 127.12.182.126:33885)
0928 02:58:25.741484 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:25.741491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:25.743069 (+ 1578us) server_negotiation.cc:407] Connection header received
0928 02:58:25.745881 (+ 2812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:25.745887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:25.745979 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:25.746100 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:25.748360 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.749346 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.752073 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.752592 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.752719 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.757144 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:25.757149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:25.757152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:25.757184 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:25.760342 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:25.761191 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:25.761197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:25.761200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:25.761327 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:25.761896 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:25.761901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:25.761903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:25.762169 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:25.762316 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:25.762963 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 02:58:25.763198 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:58:25.764506 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.742074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33885 (local address 127.12.182.67:56473)
0928 02:58:25.742884 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:25.742938 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:25.745503 (+ 2565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:25.746396 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:25.746410 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:25.747226 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:25.748159 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.748189 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.749517 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.749524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.751918 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.751937 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.752854 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.752861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.753161 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.754009 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:25.754041 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:25.756945 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:25.760482 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:25.760494 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:25.760510 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:25.761061 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:25.761455 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:25.761463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:25.761467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:25.761783 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:25.762291 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:25.762299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:25.762632 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 02:58:25.762847 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 02:58:25.763333 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":320,"threads_started":1}
I20260928 02:58:25.782703 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.769591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:34561 (local address 127.12.182.126:33885)
0928 02:58:25.769770 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:25.769776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:25.770452 (+ 676us) server_negotiation.cc:407] Connection header received
0928 02:58:25.770607 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:25.770612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:25.770672 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:25.770763 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:25.771591 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.772253 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.773497 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.774020 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.774129 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.776484 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:25.776492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:25.776497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:25.776537 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:25.779742 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:25.780447 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:25.780453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:25.780457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:25.780653 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:25.781094 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:25.781100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:25.781104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:25.781444 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:25.781587 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:25.782117 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 02:58:25.782338 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:58:25.782843 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.769659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33885 (local address 127.12.182.67:34561)
0928 02:58:25.770357 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:25.770374 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:25.770505 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:25.770795 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:25.770799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:25.771193 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:25.771452 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.771460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.772366 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.772371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.773369 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.773381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.774242 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.774249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.774385 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.775111 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:25.775132 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:25.776279 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:25.779870 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:25.779875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:25.779879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:25.780318 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:25.780805 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:25.780810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:25.780817 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:25.780980 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:25.781570 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:25.781575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:25.781807 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 02:58:25.781973 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 02:58:25.782312 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":405,"threads_started":1}
I20260928 02:58:25.786680 14490 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:34225
I20260928 02:58:25.786749 14624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:34225 every 8 connection(s)
I20260928 02:58:25.787832 14490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:58:25.789554 14625 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33885
I20260928 02:58:25.789785 14625 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:25.790351 14625 heartbeater.cc:511] Master 127.12.182.126:33885 requested a full tablet report, sending...
I20260928 02:58:25.791486 14140 ts_manager.cc:194] Registered new tserver with Master: 52ac678b40f84a0eab1100cb5bf32e1d (127.12.182.67:34225)
I20260928 02:58:25.791702 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 14490
I20260928 02:58:25.791872 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:58:25.792281 13017 external_mini_cluster.cc:1428] Setting key 351b14c6b4eef84515757c8e7355bad4
I20260928 02:58:25.792377 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:56473
I20260928 02:58:25.794905 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:58:25 dist-test-slave-k5rg krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:58:25 dist-test-slave-k5rg krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:25.835762 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.822106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.12.182.126:33885)
0928 02:58:25.822303 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:25.822309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:25.822585 (+ 276us) server_negotiation.cc:407] Connection header received
0928 02:58:25.822758 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:25.822762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:25.822830 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:25.822914 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:25.824306 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.825040 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.826749 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.827184 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.827269 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.831414 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:25.831419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:25.831422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:25.831454 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:25.833809 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:25.834311 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:25.834314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:25.834316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:25.834427 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:25.834690 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:25.834693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:25.834695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:25.834929 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:25.835090 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:25.835301 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:58:25.835454 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:58:25.835790 14347 heartbeater.cc:503] Master 127.12.182.126:33885 was elected leader, sending a full tablet report...
I20260928 02:58:25.839557 14141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:58:25.846076 14141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:25.881776 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.858845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:43591 (local address 127.0.0.1:42350)
0928 02:58:25.874391 (+ 15546us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:25.874408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:25.874533 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:25.875713 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:25.875718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:25.875742 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:25.876235 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.876243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.877588 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.877593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.879477 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.879489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.880800 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.880806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.881010 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.881068 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 02:58:25.881174 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:58:25.881339 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15439,"mutex_wait_us":111,"thread_start_us":14938,"threads_started":1}
I20260928 02:58:25.882447 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.859363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.12.182.66:43591)
0928 02:58:25.873566 (+ 14203us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:25.873575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:25.875085 (+ 1510us) server_negotiation.cc:407] Connection header received
0928 02:58:25.875196 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:25.875204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:25.875426 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:25.875604 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:25.876408 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.877447 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.879619 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.880445 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.881710 (+ 1265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.881912 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:25.881991 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:58:25.882153 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14035,"thread_start_us":13895,"threads_started":1}
I20260928 02:58:25.883104 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.857089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.12.182.65:43529)
0928 02:58:25.875049 (+ 17960us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:25.875057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:25.875084 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:58:25.875158 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:25.875162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:25.875341 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:25.875497 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:25.876360 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.877883 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.880618 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.881968 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.882096 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.882275 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:25.882741 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 02:58:25.882896 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17757,"thread_start_us":17630,"threads_started":1}
I20260928 02:58:25.883544 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.856925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43529 (local address 127.0.0.1:37064)
0928 02:58:25.872065 (+ 15140us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:25.872117 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:25.872301 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:25.875498 (+ 3197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:25.875503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:25.875540 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:25.876185 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.876199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.878017 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.878022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.880479 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.880489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.882405 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.882412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.882580 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.882611 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:58:25.882695 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:58:25.882814 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14872,"mutex_wait_us":169,"thread_start_us":15558,"threads_started":1}
I20260928 02:58:25.884202 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.857099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:34225 (local address 127.0.0.1:53852)
0928 02:58:25.874091 (+ 16992us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:25.874110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:25.874347 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:25.875425 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:25.875432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:25.875460 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:25.875950 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.875965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.878017 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.878022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.880417 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.880432 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.883404 (+ 2972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.883410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.883573 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.883604 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:58:25.883689 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:58:25.883803 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16844,"mutex_wait_us":27,"spinlock_wait_cycles":17664,"thread_start_us":17058,"threads_started":1}
I20260928 02:58:25.884961 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.857449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.12.182.67:34225)
0928 02:58:25.867001 (+ 9552us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:25.867011 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:25.874247 (+ 7236us) server_negotiation.cc:407] Connection header received
0928 02:58:25.874549 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:25.874558 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:25.874816 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:25.875031 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:25.876195 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.877850 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.880588 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.881960 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.882078 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.882487 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:25.884500 (+ 2013us) server_negotiation.cc:299] Negotiation successful
0928 02:58:25.884654 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9381,"thread_start_us":9225,"threads_started":1}
I20260928 02:58:25.886173 14416 tablet_service.cc:1467] Processing CreateTablet for tablet e43bf81124ba453fa3b0726d2e417203 (DEFAULT_TABLE table=test-table [id=a22c4a54e5804730a2800b5433ce9685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:25.886705 14277 tablet_service.cc:1467] Processing CreateTablet for tablet e43bf81124ba453fa3b0726d2e417203 (DEFAULT_TABLE table=test-table [id=a22c4a54e5804730a2800b5433ce9685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:25.887337 14416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43bf81124ba453fa3b0726d2e417203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:25.888085 14277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43bf81124ba453fa3b0726d2e417203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:25.888546 14555 tablet_service.cc:1467] Processing CreateTablet for tablet e43bf81124ba453fa3b0726d2e417203 (DEFAULT_TABLE table=test-table [id=a22c4a54e5804730a2800b5433ce9685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:25.889858 14555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43bf81124ba453fa3b0726d2e417203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:25.897686 14644 tablet_bootstrap.cc:492] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: Bootstrap starting.
I20260928 02:58:25.900425 14644 tablet_bootstrap.cc:654] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:25.901463 14644 log.cc:824] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:25.901626 14645 tablet_bootstrap.cc:492] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: Bootstrap starting.
I20260928 02:58:25.904085 14647 tablet_bootstrap.cc:492] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: Bootstrap starting.
I20260928 02:58:25.905087 14645 tablet_bootstrap.cc:654] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:25.906167 14645 log.cc:824] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:25.907158 14647 tablet_bootstrap.cc:654] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:25.907181 14644 tablet_bootstrap.cc:492] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: No bootstrap required, opened a new log
I20260928 02:58:25.907404 14644 ts_tablet_manager.cc:1397] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 02:58:25.908113 14647 log.cc:824] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:25.909025 14645 tablet_bootstrap.cc:492] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: No bootstrap required, opened a new log
I20260928 02:58:25.909292 14645 ts_tablet_manager.cc:1397] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 02:58:25.910502 14647 tablet_bootstrap.cc:492] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: No bootstrap required, opened a new log
I20260928 02:58:25.910708 14647 ts_tablet_manager.cc:1397] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260928 02:58:25.917317 14644 raft_consensus.cc:348] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } }
I20260928 02:58:25.917753 14644 raft_consensus.cc:374] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:25.917851 14644 raft_consensus.cc:729] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bacb67f7e544762a9213e1aa0a7e422, State: Initialized, Role: FOLLOWER
I20260928 02:58:25.918035 14647 raft_consensus.cc:348] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } }
I20260928 02:58:25.918381 14647 raft_consensus.cc:374] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:25.918485 14647 raft_consensus.cc:729] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ac678b40f84a0eab1100cb5bf32e1d, State: Initialized, Role: FOLLOWER
I20260928 02:58:25.918424 14644 consensus_queue.cc:260] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } }
I20260928 02:58:25.919488 14486 heartbeater.cc:503] Master 127.12.182.126:33885 was elected leader, sending a full tablet report...
I20260928 02:58:25.919288 14645 raft_consensus.cc:348] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } }
I20260928 02:58:25.919697 14645 raft_consensus.cc:374] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:25.919827 14645 raft_consensus.cc:729] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ce12d4b8224c93917ac604762f9b9e, State: Initialized, Role: FOLLOWER
I20260928 02:58:25.920152 14644 ts_tablet_manager.cc:1428] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 02:58:25.920411 14645 consensus_queue.cc:260] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } }
I20260928 02:58:25.919144 14647 consensus_queue.cc:260] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } }
I20260928 02:58:25.921919 14645 ts_tablet_manager.cc:1428] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260928 02:58:25.922816 14625 heartbeater.cc:503] Master 127.12.182.126:33885 was elected leader, sending a full tablet report...
I20260928 02:58:25.923709 14647 ts_tablet_manager.cc:1428] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: Time spent starting tablet: real 0.013s user 0.008s sys 0.001s
I20260928 02:58:25.967111 14651 raft_consensus.cc:482] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:25.967446 14651 raft_consensus.cc:504] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } }
I20260928 02:58:25.969054 14651 leader_election.cc:288] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ac678b40f84a0eab1100cb5bf32e1d (127.12.182.67:34225), 7bacb67f7e544762a9213e1aa0a7e422 (127.12.182.66:43591)
I20260928 02:58:25.975821 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.969518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:56883 (local address 127.12.182.67:34225)
0928 02:58:25.969819 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:25.969825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:25.970207 (+ 382us) server_negotiation.cc:407] Connection header received
0928 02:58:25.970745 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:25.970751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:25.970826 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:25.970910 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:25.971560 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.972222 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.974129 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.974900 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.975057 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.975127 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:25.975452 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 02:58:25.975577 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260928 02:58:25.975883 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.969433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:33243 (local address 127.12.182.66:43591)
0928 02:58:25.969666 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:25.969672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:25.970448 (+ 776us) server_negotiation.cc:407] Connection header received
0928 02:58:25.970697 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:25.970701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:25.970762 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:25.970850 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:25.971534 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.972237 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.974138 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.974945 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.975114 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.975188 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:25.975523 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 02:58:25.975642 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:58:25.975992 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.969327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:34225 (local address 127.12.182.65:56883)
0928 02:58:25.970097 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:25.970118 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:25.970339 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:25.971045 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:25.971049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:25.971088 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:25.971418 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.971427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.972345 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.972349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.973953 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.973965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.975141 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.975147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.975299 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.975325 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:58:25.975406 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:58:25.975592 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":117,"thread_start_us":851,"threads_started":1}
I20260928 02:58:25.976040 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:25.969370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:43591 (local address 127.12.182.65:33243)
0928 02:58:25.970338 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:25.970356 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:25.970556 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:25.970868 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:25.970872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:25.970932 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:25.971369 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.971383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.972364 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.972367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.973967 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:25.973982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:25.975237 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:25.975245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:25.975371 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:25.975393 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:58:25.975470 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:58:25.975586 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":4,"spinlock_wait_cycles":18176,"thread_start_us":366,"threads_started":1}
I20260928 02:58:25.976913 14436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43bf81124ba453fa3b0726d2e417203" candidate_uuid: "45ce12d4b8224c93917ac604762f9b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bacb67f7e544762a9213e1aa0a7e422" is_pre_election: true
I20260928 02:58:25.976917 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43bf81124ba453fa3b0726d2e417203" candidate_uuid: "45ce12d4b8224c93917ac604762f9b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" is_pre_election: true
I20260928 02:58:25.977437 14436 raft_consensus.cc:2454] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45ce12d4b8224c93917ac604762f9b9e in term 0.
I20260928 02:58:25.977437 14575 raft_consensus.cc:2454] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45ce12d4b8224c93917ac604762f9b9e in term 0.
I20260928 02:58:25.978152 14231 leader_election.cc:302] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45ce12d4b8224c93917ac604762f9b9e, 52ac678b40f84a0eab1100cb5bf32e1d, 7bacb67f7e544762a9213e1aa0a7e422; no voters:
I20260928 02:58:25.978554 14651 raft_consensus.cc:2781] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:25.978682 14651 raft_consensus.cc:482] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:25.978772 14651 raft_consensus.cc:3037] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:25.980873 14651 raft_consensus.cc:504] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } }
I20260928 02:58:25.981402 14651 leader_election.cc:288] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [CANDIDATE]: Term 1 election: Requested vote from peers 52ac678b40f84a0eab1100cb5bf32e1d (127.12.182.67:34225), 7bacb67f7e544762a9213e1aa0a7e422 (127.12.182.66:43591)
I20260928 02:58:25.981879 14436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43bf81124ba453fa3b0726d2e417203" candidate_uuid: "45ce12d4b8224c93917ac604762f9b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bacb67f7e544762a9213e1aa0a7e422"
I20260928 02:58:25.981879 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43bf81124ba453fa3b0726d2e417203" candidate_uuid: "45ce12d4b8224c93917ac604762f9b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ac678b40f84a0eab1100cb5bf32e1d"
I20260928 02:58:25.982085 14575 raft_consensus.cc:3037] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:25.982086 14436 raft_consensus.cc:3037] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:25.985152 14575 raft_consensus.cc:2454] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45ce12d4b8224c93917ac604762f9b9e in term 1.
I20260928 02:58:25.985152 14436 raft_consensus.cc:2454] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45ce12d4b8224c93917ac604762f9b9e in term 1.
I20260928 02:58:25.985527 14231 leader_election.cc:302] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45ce12d4b8224c93917ac604762f9b9e, 52ac678b40f84a0eab1100cb5bf32e1d, 7bacb67f7e544762a9213e1aa0a7e422; no voters:
I20260928 02:58:25.985831 14651 raft_consensus.cc:2781] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:25.986212 14651 raft_consensus.cc:686] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 1 LEADER]: Becoming Leader. State: Replica: 45ce12d4b8224c93917ac604762f9b9e, State: Running, Role: LEADER
I20260928 02:58:25.986670 14651 consensus_queue.cc:237] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } }
I20260928 02:58:25.990674 14141 catalog_manager.cc:5495] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 45ce12d4b8224c93917ac604762f9b9e (127.12.182.65). New cstate: current_term: 1 leader_uuid: "45ce12d4b8224c93917ac604762f9b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } health_report { overall_health: UNKNOWN } } }
I20260928 02:58:26.010763 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.005778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.12.182.65:43529)
0928 02:58:26.005933 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.005938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.006112 (+ 174us) server_negotiation.cc:407] Connection header received
0928 02:58:26.006368 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.006372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.006444 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.006559 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:26.006825 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.007508 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.008795 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.009243 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.009327 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.009823 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:26.010286 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:26.010390 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.010499 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.010565 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:58:26.018343 14575 raft_consensus.cc:1261] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 FOLLOWER]: Refusing update from remote peer 45ce12d4b8224c93917ac604762f9b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:26.018348 14436 raft_consensus.cc:1261] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422 [term 1 FOLLOWER]: Refusing update from remote peer 45ce12d4b8224c93917ac604762f9b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:26.019207 14655 consensus_queue.cc:1035] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:26.019502 14651 consensus_queue.cc:1035] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:58:26.039186 14626 tablet.cc:2369] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:58:26.047704 14141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:26.047928 14141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:26.051178 14141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87: Sending DeleteTablet for 3 replicas of tablet e43bf81124ba453fa3b0726d2e417203
I20260928 02:58:26.052148 14277 tablet_service.cc:1514] Processing DeleteTablet for tablet e43bf81124ba453fa3b0726d2e417203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:26 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:37064
I20260928 02:58:26.052248 14555 tablet_service.cc:1514] Processing DeleteTablet for tablet e43bf81124ba453fa3b0726d2e417203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:26 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:53852
I20260928 02:58:26.052495 14416 tablet_service.cc:1514] Processing DeleteTablet for tablet e43bf81124ba453fa3b0726d2e417203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:26 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:42350
I20260928 02:58:26.053443 14141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:58:26.054292 14141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:26.057096 14670 tablet_replica.cc:330] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: stopping tablet replica
I20260928 02:58:26.057543 14670 raft_consensus.cc:2229] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:26.058107 14670 raft_consensus.cc:2258] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:26.061538 14670 ts_tablet_manager.cc:1905] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:26.062562 14276 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff (DEFAULT_TABLE table=test-table [id=18a993d8acef4191b4d06b73c65754dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:26.063197 14276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:26.065680 14669 tablet_replica.cc:330] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: stopping tablet replica
I20260928 02:58:26.066114 14669 raft_consensus.cc:2229] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:26.066669 14669 raft_consensus.cc:2258] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:26.068895 14416 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff (DEFAULT_TABLE table=test-table [id=18a993d8acef4191b4d06b73c65754dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:26.069324 14416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:26.070480 14670 ts_tablet_manager.cc:1918] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:58:26.070645 14670 log.cc:1196] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/e43bf81124ba453fa3b0726d2e417203
I20260928 02:58:26.071242 14670 ts_tablet_manager.cc:1939] T e43bf81124ba453fa3b0726d2e417203 P 7bacb67f7e544762a9213e1aa0a7e422: Deleting consensus metadata
I20260928 02:58:26.071517 14645 tablet_bootstrap.cc:492] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: Bootstrap starting.
I20260928 02:58:26.073158 14645 tablet_bootstrap.cc:654] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:26.073096 14125 catalog_manager.cc:4841] TS 7bacb67f7e544762a9213e1aa0a7e422 (127.12.182.66:43591): tablet e43bf81124ba453fa3b0726d2e417203 (table test-table [id=a22c4a54e5804730a2800b5433ce9685]) successfully deleted
I20260928 02:58:26.075598 14645 tablet_bootstrap.cc:492] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: No bootstrap required, opened a new log
I20260928 02:58:26.075786 14645 ts_tablet_manager.cc:1397] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 02:58:26.076320 14555 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff (DEFAULT_TABLE table=test-table [id=18a993d8acef4191b4d06b73c65754dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:26.076645 14645 raft_consensus.cc:348] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } }
I20260928 02:58:26.076908 14645 raft_consensus.cc:374] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:26.077045 14645 raft_consensus.cc:729] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ce12d4b8224c93917ac604762f9b9e, State: Initialized, Role: FOLLOWER
I20260928 02:58:26.077301 14555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:26.077381 14645 consensus_queue.cc:260] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } }
I20260928 02:58:26.077828 14644 tablet_bootstrap.cc:492] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: Bootstrap starting.
W20260928 02:58:26.078246 14348 tablet.cc:2369] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:58:26.079276 14644 tablet_bootstrap.cc:654] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:26.079387 14645 ts_tablet_manager.cc:1428] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:58:26.087730 14647 tablet_bootstrap.cc:492] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d: Bootstrap starting.
I20260928 02:58:26.087992 14668 tablet_replica.cc:330] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: stopping tablet replica
I20260928 02:58:26.088430 14668 raft_consensus.cc:2229] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:58:26.088526 14669 ts_tablet_manager.cc:1905] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:26.089042 14668 raft_consensus.cc:2258] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:26.089280 14647 tablet_bootstrap.cc:654] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:26.092784 14668 ts_tablet_manager.cc:1905] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:26.097569 14669 ts_tablet_manager.cc:1918] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:58:26.097826 14669 log.cc:1196] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/e43bf81124ba453fa3b0726d2e417203
I20260928 02:58:26.098405 14669 ts_tablet_manager.cc:1939] T e43bf81124ba453fa3b0726d2e417203 P 52ac678b40f84a0eab1100cb5bf32e1d: Deleting consensus metadata
I20260928 02:58:26.099504 14644 tablet_bootstrap.cc:492] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: No bootstrap required, opened a new log
I20260928 02:58:26.099663 14644 ts_tablet_manager.cc:1397] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.007s
I20260928 02:58:26.100426 14128 catalog_manager.cc:4841] TS 52ac678b40f84a0eab1100cb5bf32e1d (127.12.182.67:34225): tablet e43bf81124ba453fa3b0726d2e417203 (table test-table [id=a22c4a54e5804730a2800b5433ce9685]) successfully deleted
I20260928 02:58:26.100523 14644 raft_consensus.cc:348] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } }
I20260928 02:58:26.100760 14644 raft_consensus.cc:374] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:26.100853 14644 raft_consensus.cc:729] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bacb67f7e544762a9213e1aa0a7e422, State: Initialized, Role: FOLLOWER
I20260928 02:58:26.101110 14644 consensus_queue.cc:260] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } }
I20260928 02:58:26.101704 14644 ts_tablet_manager.cc:1428] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 02:58:26.103816 14647 tablet_bootstrap.cc:492] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d: No bootstrap required, opened a new log
I20260928 02:58:26.104059 14647 ts_tablet_manager.cc:1397] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.008s
I20260928 02:58:26.104923 14647 raft_consensus.cc:348] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } }
I20260928 02:58:26.105310 14647 raft_consensus.cc:374] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:26.105451 14647 raft_consensus.cc:729] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ac678b40f84a0eab1100cb5bf32e1d, State: Initialized, Role: FOLLOWER
I20260928 02:58:26.105712 14668 ts_tablet_manager.cc:1918] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:58:26.105880 14668 log.cc:1196] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/e43bf81124ba453fa3b0726d2e417203
I20260928 02:58:26.105752 14647 consensus_queue.cc:260] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } }
I20260928 02:58:26.106429 14668 ts_tablet_manager.cc:1939] T e43bf81124ba453fa3b0726d2e417203 P 45ce12d4b8224c93917ac604762f9b9e: Deleting consensus metadata
I20260928 02:58:26.106539 14647 ts_tablet_manager.cc:1428] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:58:26.108081 14126 catalog_manager.cc:4841] TS 45ce12d4b8224c93917ac604762f9b9e (127.12.182.65:43529): tablet e43bf81124ba453fa3b0726d2e417203 (table test-table [id=a22c4a54e5804730a2800b5433ce9685]) successfully deleted
I20260928 02:58:26.172531 14652 raft_consensus.cc:482] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:26.172745 14652 raft_consensus.cc:504] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } }
I20260928 02:58:26.174422 14652 leader_election.cc:288] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45ce12d4b8224c93917ac604762f9b9e (127.12.182.65:43529), 7bacb67f7e544762a9213e1aa0a7e422 (127.12.182.66:43591)
I20260928 02:58:26.181370 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.174815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:43591 (local address 127.12.182.67:53397)
0928 02:58:26.175276 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:26.175294 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:26.175408 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:26.175982 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:26.175987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:26.176020 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:26.176212 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:26.176224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.177479 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.177484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:26.179582 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:26.179600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.180692 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.180697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:26.180867 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.180892 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:58:26.180972 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:58:26.181099 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320}
I20260928 02:58:26.181397 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.175369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:53397 (local address 127.12.182.66:43591)
0928 02:58:26.175556 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.175562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.175583 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:58:26.175663 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.175668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.175745 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.175833 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:26.176398 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.177333 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.179753 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.180539 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.180690 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.180764 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.181022 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.181140 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:58:26.182054 14436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc824ed0f2c4ffd8c65ccac19f0e9ff" candidate_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bacb67f7e544762a9213e1aa0a7e422" is_pre_election: true
I20260928 02:58:26.182302 14436 raft_consensus.cc:2454] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ac678b40f84a0eab1100cb5bf32e1d in term 0.
I20260928 02:58:26.182821 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.174914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:59827 (local address 127.12.182.65:43529)
0928 02:58:26.175125 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.175132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.176118 (+ 986us) server_negotiation.cc:407] Connection header received
0928 02:58:26.176366 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.176371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.176484 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.176545 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:26.177253 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.178108 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.179847 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.181081 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.181248 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.181318 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.182377 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.182510 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:58:26.182833 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.174828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43529 (local address 127.12.182.67:59827)
0928 02:58:26.176009 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:26.176041 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:26.176220 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:26.176728 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:26.176732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:26.176755 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:26.177080 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:26.177091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.178237 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.178242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:26.179633 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:26.179645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.181970 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.181976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:26.182163 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.182196 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:58:26.182328 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 02:58:26.182452 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":226,"thread_start_us":625,"threads_started":1}
I20260928 02:58:26.182942 14506 leader_election.cc:302] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ac678b40f84a0eab1100cb5bf32e1d, 7bacb67f7e544762a9213e1aa0a7e422; no voters:
I20260928 02:58:26.183341 14652 raft_consensus.cc:2781] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:26.183444 14652 raft_consensus.cc:482] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:26.183532 14652 raft_consensus.cc:3037] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:26.183774 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc824ed0f2c4ffd8c65ccac19f0e9ff" candidate_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45ce12d4b8224c93917ac604762f9b9e" is_pre_election: true
I20260928 02:58:26.184257 14297 raft_consensus.cc:2454] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ac678b40f84a0eab1100cb5bf32e1d in term 0.
I20260928 02:58:26.185839 14652 raft_consensus.cc:504] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } }
I20260928 02:58:26.186369 14652 leader_election.cc:288] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [CANDIDATE]: Term 1 election: Requested vote from peers 45ce12d4b8224c93917ac604762f9b9e (127.12.182.65:43529), 7bacb67f7e544762a9213e1aa0a7e422 (127.12.182.66:43591)
I20260928 02:58:26.186862 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc824ed0f2c4ffd8c65ccac19f0e9ff" candidate_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45ce12d4b8224c93917ac604762f9b9e"
I20260928 02:58:26.186854 14436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc824ed0f2c4ffd8c65ccac19f0e9ff" candidate_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bacb67f7e544762a9213e1aa0a7e422"
I20260928 02:58:26.187063 14297 raft_consensus.cc:3037] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:26.187054 14436 raft_consensus.cc:3037] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:26.189982 14436 raft_consensus.cc:2454] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ac678b40f84a0eab1100cb5bf32e1d in term 1.
I20260928 02:58:26.190196 14297 raft_consensus.cc:2454] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ac678b40f84a0eab1100cb5bf32e1d in term 1.
I20260928 02:58:26.190378 14506 leader_election.cc:302] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ac678b40f84a0eab1100cb5bf32e1d, 7bacb67f7e544762a9213e1aa0a7e422; no voters:
I20260928 02:58:26.190662 14652 raft_consensus.cc:2781] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:26.191109 14652 raft_consensus.cc:686] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 LEADER]: Becoming Leader. State: Replica: 52ac678b40f84a0eab1100cb5bf32e1d, State: Running, Role: LEADER
I20260928 02:58:26.191629 14652 consensus_queue.cc:237] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } }
I20260928 02:58:26.195083 14139 catalog_manager.cc:5495] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ac678b40f84a0eab1100cb5bf32e1d (127.12.182.67). New cstate: current_term: 1 leader_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } health_report { overall_health: HEALTHY } } }
Sep 28 02:58:26 dist-test-slave-k5rg krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564304, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:26.250277 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.236058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33885 (local address 127.0.0.1:54120)
0928 02:58:26.236688 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:26.236708 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:26.236857 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:26.237151 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:26.237159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:26.238068 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:26.238401 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:26.238413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.239412 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.239417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:26.240880 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:26.240895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.241596 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.241636 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:26.241797 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.242460 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:26.242480 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:26.245079 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:26.247856 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:26.247868 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:26.247876 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:26.248281 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:26.248616 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:26.248621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:26.248624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:26.248742 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:26.249215 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:26.249222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:26.249376 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 02:58:26.249518 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:58:26.249804 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":328,"threads_started":1}
I20260928 02:58:26.250298 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.236034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.12.182.126:33885)
0928 02:58:26.236233 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.236239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.236796 (+ 557us) server_negotiation.cc:407] Connection header received
0928 02:58:26.236964 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.236967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.237026 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.237102 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:26.238540 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.239282 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.241011 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.241430 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.241514 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.245241 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:26.245245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:26.245248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:26.245277 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:26.247716 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.248394 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.248398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.248400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.248521 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.248832 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.248836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.248839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.249091 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:26.249219 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.249650 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.249940 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":66}
I20260928 02:58:26.255105 14139 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:54120:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:58:26.256040 14139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:26.262588 14416 tablet_service.cc:1467] Processing CreateTablet for tablet cffdff2990df4e37adecaed6851d1758 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0050ea523ffb479eb74da530acf42034]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:58:26.263134 14416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffdff2990df4e37adecaed6851d1758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:26.263264 14277 tablet_service.cc:1467] Processing CreateTablet for tablet cffdff2990df4e37adecaed6851d1758 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0050ea523ffb479eb74da530acf42034]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:58:26.263260 14555 tablet_service.cc:1467] Processing CreateTablet for tablet cffdff2990df4e37adecaed6851d1758 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0050ea523ffb479eb74da530acf42034]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:58:26.263761 14277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffdff2990df4e37adecaed6851d1758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:26.263756 14555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffdff2990df4e37adecaed6851d1758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:26.270476 14644 tablet_bootstrap.cc:492] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: Bootstrap starting.
I20260928 02:58:26.270599 14647 tablet_bootstrap.cc:492] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: Bootstrap starting.
I20260928 02:58:26.272048 14647 tablet_bootstrap.cc:654] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:26.272094 14644 tablet_bootstrap.cc:654] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:26.273217 14645 tablet_bootstrap.cc:492] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: Bootstrap starting.
I20260928 02:58:26.274842 14645 tablet_bootstrap.cc:654] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:26.282400 14647 tablet_bootstrap.cc:492] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: No bootstrap required, opened a new log
I20260928 02:58:26.282579 14647 ts_tablet_manager.cc:1397] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260928 02:58:26.282950 14644 tablet_bootstrap.cc:492] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: No bootstrap required, opened a new log
I20260928 02:58:26.283164 14644 ts_tablet_manager.cc:1397] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260928 02:58:26.283365 14647 raft_consensus.cc:348] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } }
I20260928 02:58:26.283617 14647 raft_consensus.cc:374] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:26.283692 14647 raft_consensus.cc:729] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ac678b40f84a0eab1100cb5bf32e1d, State: Initialized, Role: FOLLOWER
I20260928 02:58:26.284116 14644 raft_consensus.cc:348] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } }
I20260928 02:58:26.284323 14647 consensus_queue.cc:260] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } }
I20260928 02:58:26.284468 14644 raft_consensus.cc:374] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:26.284720 14644 raft_consensus.cc:729] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bacb67f7e544762a9213e1aa0a7e422, State: Initialized, Role: FOLLOWER
I20260928 02:58:26.284775 14652 tablet_replica.cc:439] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } }
I20260928 02:58:26.285058 14652 tablet_replica.cc:442] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:26.285080 14644 consensus_queue.cc:260] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } }
I20260928 02:58:26.285596 14650 tablet_replica.cc:439] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } }
I20260928 02:58:26.285948 14650 tablet_replica.cc:442] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:26.286976 14644 ts_tablet_manager.cc:1428] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260928 02:58:26.288141 14647 ts_tablet_manager.cc:1428] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260928 02:58:26.290052 14645 tablet_bootstrap.cc:492] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: No bootstrap required, opened a new log
I20260928 02:58:26.290242 14645 ts_tablet_manager.cc:1397] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260928 02:58:26.290854 14645 raft_consensus.cc:348] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } }
I20260928 02:58:26.291136 14645 raft_consensus.cc:374] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:26.291234 14645 raft_consensus.cc:729] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ce12d4b8224c93917ac604762f9b9e, State: Initialized, Role: FOLLOWER
I20260928 02:58:26.291469 14645 consensus_queue.cc:260] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } }
I20260928 02:58:26.292098 14645 ts_tablet_manager.cc:1428] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:58:26.291941 14655 tablet_replica.cc:439] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } }
I20260928 02:58:26.292250 14655 tablet_replica.cc:442] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 02:58:26.310209 14487 tablet.cc:2369] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:58:26.513727 14652 raft_consensus.cc:482] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:26.513943 14652 raft_consensus.cc:504] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } }
I20260928 02:58:26.514573 14652 leader_election.cc:288] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bacb67f7e544762a9213e1aa0a7e422 (127.12.182.66:43591), 45ce12d4b8224c93917ac604762f9b9e (127.12.182.65:43529)
I20260928 02:58:26.515129 14436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdff2990df4e37adecaed6851d1758" candidate_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bacb67f7e544762a9213e1aa0a7e422" is_pre_election: true
I20260928 02:58:26.515143 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdff2990df4e37adecaed6851d1758" candidate_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45ce12d4b8224c93917ac604762f9b9e" is_pre_election: true
I20260928 02:58:26.515364 14436 raft_consensus.cc:2454] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ac678b40f84a0eab1100cb5bf32e1d in term 0.
I20260928 02:58:26.515388 14297 raft_consensus.cc:2454] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ac678b40f84a0eab1100cb5bf32e1d in term 0.
I20260928 02:58:26.515784 14506 leader_election.cc:302] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ac678b40f84a0eab1100cb5bf32e1d, 7bacb67f7e544762a9213e1aa0a7e422; no voters:
I20260928 02:58:26.516044 14652 raft_consensus.cc:2781] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:26.516130 14652 raft_consensus.cc:482] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:26.516237 14652 raft_consensus.cc:3037] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:26.518162 14652 raft_consensus.cc:504] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } }
I20260928 02:58:26.518607 14652 leader_election.cc:288] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [CANDIDATE]: Term 1 election: Requested vote from peers 7bacb67f7e544762a9213e1aa0a7e422 (127.12.182.66:43591), 45ce12d4b8224c93917ac604762f9b9e (127.12.182.65:43529)
I20260928 02:58:26.518978 14436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdff2990df4e37adecaed6851d1758" candidate_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bacb67f7e544762a9213e1aa0a7e422"
I20260928 02:58:26.519089 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdff2990df4e37adecaed6851d1758" candidate_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45ce12d4b8224c93917ac604762f9b9e"
I20260928 02:58:26.519192 14436 raft_consensus.cc:3037] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:26.519287 14297 raft_consensus.cc:3037] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:26.521973 14436 raft_consensus.cc:2454] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ac678b40f84a0eab1100cb5bf32e1d in term 1.
I20260928 02:58:26.521973 14297 raft_consensus.cc:2454] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ac678b40f84a0eab1100cb5bf32e1d in term 1.
I20260928 02:58:26.522317 14506 leader_election.cc:302] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ac678b40f84a0eab1100cb5bf32e1d, 7bacb67f7e544762a9213e1aa0a7e422; no voters:
I20260928 02:58:26.522538 14652 raft_consensus.cc:2781] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:26.522689 14652 raft_consensus.cc:686] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 LEADER]: Becoming Leader. State: Replica: 52ac678b40f84a0eab1100cb5bf32e1d, State: Running, Role: LEADER
I20260928 02:58:26.522924 14652 consensus_queue.cc:237] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } }
I20260928 02:58:26.524312 14675 tablet_replica.cc:439] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: TxnStatusTablet state changed. Reason: New leader 52ac678b40f84a0eab1100cb5bf32e1d. Latest consensus state: current_term: 1 leader_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } }
I20260928 02:58:26.524560 14675 tablet_replica.cc:442] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:58:26.525079 14687 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:58:26.525256 14687 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:58:26.525621 14141 catalog_manager.cc:5495] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ac678b40f84a0eab1100cb5bf32e1d (127.12.182.67). New cstate: current_term: 1 leader_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } health_report { overall_health: UNKNOWN } } }
I20260928 02:58:26.574879 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.568776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.12.182.67:34225)
0928 02:58:26.569271 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.569276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.569292 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:58:26.569370 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.569373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.569445 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.569575 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:26.569994 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.570779 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.572611 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.573096 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.573192 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.573711 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:26.574284 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:26.574424 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.574559 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.574621 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":243,"threads_started":1}
I20260928 02:58:26.575003 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.568661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:34225 (local address 127.0.0.1:53860)
0928 02:58:26.568894 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:26.568915 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:26.569065 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:26.569640 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:26.569645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:26.569658 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:58:26.569850 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:26.569860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.570894 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.570900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:26.572471 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:26.572487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.573250 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.573257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:26.573432 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.573498 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:58:26.574413 (+ 915us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:58:26.574439 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:58:26.574528 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:58:26.574675 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:58:26.578058 14675 consensus_queue.cc:1035] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:26.579696 14297 raft_consensus.cc:1261] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e [term 1 FOLLOWER]: Refusing update from remote peer 52ac678b40f84a0eab1100cb5bf32e1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:26.580364 14435 raft_consensus.cc:1261] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422 [term 1 FOLLOWER]: Refusing update from remote peer 52ac678b40f84a0eab1100cb5bf32e1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:26.581012 14652 consensus_queue.cc:1035] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:26.582209 14652 consensus_queue.cc:1035] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:26.599077 14655 tablet_replica.cc:439] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: TxnStatusTablet state changed. Reason: New leader 52ac678b40f84a0eab1100cb5bf32e1d. Latest consensus state: current_term: 1 leader_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } }
I20260928 02:58:26.599344 14655 tablet_replica.cc:442] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:26.599829 14675 consensus_queue.cc:1035] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:26.605891 14655 tablet_replica.cc:439] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } }
I20260928 02:58:26.606148 14655 tablet_replica.cc:442] T cffdff2990df4e37adecaed6851d1758 P 45ce12d4b8224c93917ac604762f9b9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:26.632313 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.618500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.12.182.67:34225)
0928 02:58:26.618641 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.618646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.626045 (+ 7399us) server_negotiation.cc:407] Connection header received
0928 02:58:26.626115 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.626120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.626201 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.626276 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:26.626949 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.628028 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.629895 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.630606 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.630702 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.631278 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:26.631408 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:26.631490 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.631911 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.631999 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:58:26.634652 14650 tablet_replica.cc:439] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: TxnStatusTablet state changed. Reason: New leader 52ac678b40f84a0eab1100cb5bf32e1d. Latest consensus state: current_term: 1 leader_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } }
I20260928 02:58:26.634889 14650 tablet_replica.cc:442] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:26.636950 14695 mvcc.cc:205] Tried to move back new op lower bound from 7334151399741566976 to 7334151399519842304. Current Snapshot: MvccSnapshot[applied={T|T < 7334151399741566976}]
I20260928 02:58:26.639406 14675 tablet_replica.cc:439] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } }
I20260928 02:58:26.639690 14675 tablet_replica.cc:442] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:58:26.639937 14675 tablet_replica.cc:439] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } }
I20260928 02:58:26.640129 14675 tablet_replica.cc:442] T cffdff2990df4e37adecaed6851d1758 P 52ac678b40f84a0eab1100cb5bf32e1d: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:58:26.640306 14691 mvcc.cc:205] Tried to move back new op lower bound from 7334151399741566976 to 7334151399519842304. Current Snapshot: MvccSnapshot[applied={T|T < 7334151399741566976 or (T in {7334151399741566976})}]
I20260928 02:58:26.640954 14650 tablet_replica.cc:439] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bacb67f7e544762a9213e1aa0a7e422" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43591 } } peers { permanent_uuid: "52ac678b40f84a0eab1100cb5bf32e1d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34225 } } peers { permanent_uuid: "45ce12d4b8224c93917ac604762f9b9e" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43529 } } }
I20260928 02:58:26.641187 14650 tablet_replica.cc:442] T cffdff2990df4e37adecaed6851d1758 P 7bacb67f7e544762a9213e1aa0a7e422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:26.652377 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.646401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:49187 (local address 127.12.182.67:34225)
0928 02:58:26.646608 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.646613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.646968 (+ 355us) server_negotiation.cc:407] Connection header received
0928 02:58:26.647142 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.647145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.647216 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.647292 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:26.647823 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.648648 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.650676 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.651159 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.651253 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.651738 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:26.651826 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:26.651932 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.652058 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.652120 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":49}
I20260928 02:58:26.652661 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.646323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:34225 (local address 127.12.182.67:49187)
0928 02:58:26.646878 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:26.646896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:26.647025 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:26.647324 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:26.647329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:26.647342 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:58:26.647698 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:26.647709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.648758 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.648764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:26.650538 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:26.650553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.651334 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.651340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:26.651503 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.651573 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:58:26.651952 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:58:26.651962 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:58:26.652047 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:58:26.652192 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260928 02:58:26.711045 14139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:26.711268 14139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:26.714041 14139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0134da097ddd44f6bc657fbe83068d87: Sending DeleteTablet for 3 replicas of tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff
I20260928 02:58:26.714865 14277 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:26 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:37064
I20260928 02:58:26.715006 14555 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:26 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:53852
I20260928 02:58:26.715304 14416 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:26 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:42350
I20260928 02:58:26.715687 14713 tablet_replica.cc:330] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d: stopping tablet replica
I20260928 02:58:26.715926 14713 raft_consensus.cc:2229] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:58:26.716626 14713 raft_consensus.cc:2258] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:26.717736 14713 ts_tablet_manager.cc:1905] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:26.723869 14714 tablet_replica.cc:330] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: stopping tablet replica
I20260928 02:58:26.724087 14714 raft_consensus.cc:2229] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:26.724313 14714 raft_consensus.cc:2258] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:26.725481 14713 ts_tablet_manager.cc:1918] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:58:26.725984 14713 log.cc:1196] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/5bc824ed0f2c4ffd8c65ccac19f0e9ff
I20260928 02:58:26.726526 14713 ts_tablet_manager.cc:1939] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 52ac678b40f84a0eab1100cb5bf32e1d: Deleting consensus metadata
I20260928 02:58:26.727597 14128 catalog_manager.cc:4841] TS 52ac678b40f84a0eab1100cb5bf32e1d (127.12.182.67:34225): tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff (table test-table [id=18a993d8acef4191b4d06b73c65754dd]) successfully deleted
I20260928 02:58:26.729959 14712 tablet_replica.cc:330] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: stopping tablet replica
I20260928 02:58:26.730223 14712 raft_consensus.cc:2229] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:26.730523 14712 raft_consensus.cc:2258] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:26.731513 14712 ts_tablet_manager.cc:1905] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:26.733459 14714 ts_tablet_manager.cc:1905] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:26.738335 14712 ts_tablet_manager.cc:1918] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:58:26.738461 14712 log.cc:1196] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/5bc824ed0f2c4ffd8c65ccac19f0e9ff
I20260928 02:58:26.739213 14712 ts_tablet_manager.cc:1939] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 45ce12d4b8224c93917ac604762f9b9e: Deleting consensus metadata
I20260928 02:58:26.740190 14714 ts_tablet_manager.cc:1918] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:58:26.740175 14126 catalog_manager.cc:4841] TS 45ce12d4b8224c93917ac604762f9b9e (127.12.182.65:43529): tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff (table test-table [id=18a993d8acef4191b4d06b73c65754dd]) successfully deleted
I20260928 02:58:26.740321 14714 log.cc:1196] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/5bc824ed0f2c4ffd8c65ccac19f0e9ff
I20260928 02:58:26.740738 14714 ts_tablet_manager.cc:1939] T 5bc824ed0f2c4ffd8c65ccac19f0e9ff P 7bacb67f7e544762a9213e1aa0a7e422: Deleting consensus metadata
I20260928 02:58:26.741709 14125 catalog_manager.cc:4841] TS 7bacb67f7e544762a9213e1aa0a7e422 (127.12.182.66:43591): tablet 5bc824ed0f2c4ffd8c65ccac19f0e9ff (table test-table [id=18a993d8acef4191b4d06b73c65754dd]) successfully deleted
Sep 28 02:58:26 dist-test-slave-k5rg krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:58:26.752310 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.725648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.12.182.65:43529)
0928 02:58:26.737843 (+ 12195us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.737849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.737869 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:58:26.737923 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.737928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.738018 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.740236 (+ 2218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:26.740510 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.741517 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.742825 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.743208 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.743307 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.746950 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:26.746961 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:26.746975 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:26.747008 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:26.749713 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.750302 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.750309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.750311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.750446 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.750805 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.750810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.750814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.751209 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:26.751458 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.751787 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.751999 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12072,"thread_start_us":12447,"threads_started":1}
W20260928 02:58:26.753013 14342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37078
I20260928 02:58:26.766948 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.755605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.12.182.126:33885)
0928 02:58:26.756265 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.756269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.756283 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:58:26.756326 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.756330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.756384 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.756480 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:26.757578 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.758437 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.759580 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.759982 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.760086 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.762121 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:26.762126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:26.762129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:26.762159 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:26.764535 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.765093 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.765098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.765100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.765226 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.765544 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.765548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.765551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.765858 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:26.766083 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.766380 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.766595 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":469,"threads_started":1}
W20260928 02:58:26.767491 14139 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54130
I20260928 02:58:26.769157 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14212
I20260928 02:58:26.786057 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.770529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.12.182.65:43529)
0928 02:58:26.770706 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.770712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.774234 (+ 3522us) server_negotiation.cc:407] Connection header received
0928 02:58:26.774403 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.774407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.774471 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.774545 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:26.775925 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.776604 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.778317 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.778742 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.778845 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.781275 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:26.781280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:26.781283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:26.781315 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:26.783681 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.784361 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.784364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.784367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.784491 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.784830 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.784834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.784837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.785058 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:26.785245 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.785530 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.785744 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 02:58:26.789928 14342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37090
W20260928 02:58:26.791225 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:58:26.791421 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 14212
I20260928 02:58:26.809638 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14351
Sep 28 02:58:26 dist-test-slave-k5rg krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 02:58:26.824826 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.810380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.12.182.66:43591)
0928 02:58:26.811306 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.811312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.811330 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:58:26.811389 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.811393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.811468 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.812675 (+ 1207us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:26.812787 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.813431 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.814827 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.815267 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.815373 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.819103 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:26.819116 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:26.819129 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:26.819163 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:26.821806 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.822434 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.822441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.822443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.822565 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.822953 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.822958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.822962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.823300 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:26.823632 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.824400 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.824565 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":724,"threads_started":1}
W20260928 02:58:26.825496 14481 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42356
W20260928 02:58:26.826120 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:58:26.826272 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 14351
I20260928 02:58:26.840579 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14490
Sep 28 02:58:26 dist-test-slave-k5rg krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 02:58:26.855166 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:26.841282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.12.182.67:34225)
0928 02:58:26.841469 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:26.841474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:26.841697 (+ 223us) server_negotiation.cc:407] Connection header received
0928 02:58:26.841871 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:26.841877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:26.841943 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:26.842029 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:26.843158 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.843967 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.845141 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:26.845661 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:26.845788 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:26.849568 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:26.849579 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:26.849594 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:26.849666 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:26.852577 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.853168 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.853177 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.853180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.853325 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:26.853697 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:26.853702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:26.853706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:26.854087 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:26.854371 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:26.854647 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 02:58:26.854876 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 02:58:26.855801 14620 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53888
W20260928 02:58:26.856526 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14490: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:58:26.856652 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 14490
I20260928 02:58:26.871829 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14109
I20260928 02:58:26.872792 14181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:26.985327 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 14109
2026-09-28T02:58:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4274 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:27 dist-test-slave-k5rg krb5kdc[14726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:27 dist-test-slave-k5rg krb5kdc[14726](info): set up 2 sockets
Sep 28 02:58:27 dist-test-slave-k5rg krb5kdc[14726](info): commencing operation
krb5kdc: starting...
W20260928 02:58:27.965641 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:58:28 dist-test-slave-k5rg krb5kdc[14726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:58:28.129208 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:34775
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:34775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:28.431864 14739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:28.432174 14739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:28.432296 14739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:28.439693 14739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:28.439805 14739 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:58:28.439847 14739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:28.439929 14739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:28.439997 14739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:28.440088 14739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:28.450045 14739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:34775
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:34775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:28.451315 14739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:28.452620 14739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:28.463877 14744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:28.463851 14745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:28.464408 14747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:28.464672 14739 server_base.cc:1034] running on GCE node
I20260928 02:58:28.465389 14739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:28.466749 14739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:28.467949 14739 hybrid_clock.cc:648] HybridClock initialized: now 1790564308467882 us; error 52 us; skew 500 ppm
I20260928 02:58:28.468247 14739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:28 dist-test-slave-k5rg krb5kdc[14726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564308, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:28.471247 14739 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:58:28.472668 14739 webserver.cc:460] Webserver started at http://127.12.182.126:40529/ using document root <none> and password file <none>
I20260928 02:58:28.473371 14739 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:28.473459 14739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:28.473783 14739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:28.476666 14739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "ca9ee76bce504f80b214e312c22c421a"
format_stamp: "Formatted at 2026-09-28 02:58:28 on dist-test-slave-k5rg"
server_key: "f7179c0477088c42c1890c8e8ebb721d"
server_key_iv: "6bed6c0d334bc051bd9e2b4bcc6fc3be"
server_key_version: "encryptionkey@0"
I20260928 02:58:28.477420 14739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "ca9ee76bce504f80b214e312c22c421a"
format_stamp: "Formatted at 2026-09-28 02:58:28 on dist-test-slave-k5rg"
server_key: "f7179c0477088c42c1890c8e8ebb721d"
server_key_iv: "6bed6c0d334bc051bd9e2b4bcc6fc3be"
server_key_version: "encryptionkey@0"
I20260928 02:58:28.483500 14739 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260928 02:58:28.488165 14754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:28.489712 14739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:58:28.489895 14739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "ca9ee76bce504f80b214e312c22c421a"
format_stamp: "Formatted at 2026-09-28 02:58:28 on dist-test-slave-k5rg"
server_key: "f7179c0477088c42c1890c8e8ebb721d"
server_key_iv: "6bed6c0d334bc051bd9e2b4bcc6fc3be"
server_key_version: "encryptionkey@0"
I20260928 02:58:28.490110 14739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:28.508107 14739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:28.512020 14739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:28.512262 14739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:28.535485 14739 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:34775
I20260928 02:58:28.535506 14815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:34775 every 8 connection(s)
I20260928 02:58:28.536779 14739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:28.540462 14816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:28.545665 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 14739
I20260928 02:58:28.545845 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:28.546446 13017 external_mini_cluster.cc:1428] Setting key dd3db62e5d22a668eba326a4a4915837
I20260928 02:58:28.551656 14816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a: Bootstrap starting.
I20260928 02:58:28.555225 14816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:28.556279 14816 log.cc:824] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:28.559020 14816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a: No bootstrap required, opened a new log
Sep 28 02:58:28 dist-test-slave-k5rg krb5kdc[14726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:28.570405 14816 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9ee76bce504f80b214e312c22c421a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34775 } }
I20260928 02:58:28.570931 14816 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:28.571038 14816 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9ee76bce504f80b214e312c22c421a, State: Initialized, Role: FOLLOWER
I20260928 02:58:28.571732 14816 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9ee76bce504f80b214e312c22c421a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34775 } }
I20260928 02:58:28.572014 14816 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:28.572150 14816 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:28.572317 14816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:28.573026 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:28.548314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.12.182.126:34775)
0928 02:58:28.549097 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:28.549117 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:28.549165 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:58:28.551421 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:28.551449 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:28.551924 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:28.552343 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:28.553573 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:28.555470 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:28.556715 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:28.557538 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:28.557739 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:28.561378 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:28.561393 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:28.561413 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:28.561465 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:28.567960 (+ 6495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:28.569078 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:28.569088 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:28.569099 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:28.569349 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:28.569849 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:28.569854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:28.569857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:28.570632 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:28.570913 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:28.571423 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 02:58:28.571867 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":330,"threads_started":1}
I20260928 02:58:28.575567 14816 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9ee76bce504f80b214e312c22c421a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34775 } }
I20260928 02:58:28.576116 14816 leader_election.cc:302] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca9ee76bce504f80b214e312c22c421a; no voters:
I20260928 02:58:28.576684 14816 leader_election.cc:288] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:28.576802 14821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:28.577426 14821 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [term 1 LEADER]: Becoming Leader. State: Replica: ca9ee76bce504f80b214e312c22c421a, State: Running, Role: LEADER
I20260928 02:58:28.577965 14821 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9ee76bce504f80b214e312c22c421a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34775 } }
I20260928 02:58:28.578568 14816 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:28.582291 14823 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca9ee76bce504f80b214e312c22c421a. Latest consensus state: current_term: 1 leader_uuid: "ca9ee76bce504f80b214e312c22c421a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9ee76bce504f80b214e312c22c421a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34775 } } }
I20260928 02:58:28.583063 14823 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:28.582986 14822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca9ee76bce504f80b214e312c22c421a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9ee76bce504f80b214e312c22c421a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34775 } } }
I20260928 02:58:28.583297 14822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:28.584587 14827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:28.589090 14827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:28.596961 14827 catalog_manager.cc:1347] Generated new cluster ID: daf23abc805049d8bc12237926081d46
I20260928 02:58:28.597060 14827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:28.625341 14827 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:28.627241 14827 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:28.635285 14827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a: Generated new TSK 0
I20260928 02:58:28.636232 14827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:58:28.678442 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:34775
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:29.008154 14842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:29.008420 14842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:29.008550 14842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:29.015760 14842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:29.015863 14842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:29.016018 14842 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:58:29.016088 14842 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:58:29.016170 14842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:58:29.025969 14842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.182.126:34775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:29.027216 14842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:29.028517 14842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:29.041047 14847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:29.041081 14848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:29.040970 14850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:29.041954 14842 server_base.cc:1034] running on GCE node
I20260928 02:58:29.042552 14842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:29.043468 14842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:29.044663 14842 hybrid_clock.cc:648] HybridClock initialized: now 1790564309044611 us; error 40 us; skew 500 ppm
I20260928 02:58:29.044975 14842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:29 dist-test-slave-k5rg krb5kdc[14726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564309, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:29.048172 14842 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:58:29.049626 14842 webserver.cc:460] Webserver started at http://127.12.182.65:46551/ using document root <none> and password file <none>
I20260928 02:58:29.050340 14842 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:29.050464 14842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:29.050756 14842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:29.053682 14842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "a950b7016e0d49ae874677d2ffb09265"
format_stamp: "Formatted at 2026-09-28 02:58:29 on dist-test-slave-k5rg"
server_key: "9981558bc852d17d33dbf8776b92afd9"
server_key_iv: "984c4baf808832974961c78aa90401e6"
server_key_version: "encryptionkey@0"
I20260928 02:58:29.054436 14842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "a950b7016e0d49ae874677d2ffb09265"
format_stamp: "Formatted at 2026-09-28 02:58:29 on dist-test-slave-k5rg"
server_key: "9981558bc852d17d33dbf8776b92afd9"
server_key_iv: "984c4baf808832974961c78aa90401e6"
server_key_version: "encryptionkey@0"
I20260928 02:58:29.059432 14842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:58:29.062881 14857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:29.064368 14842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:29.064532 14842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "a950b7016e0d49ae874677d2ffb09265"
format_stamp: "Formatted at 2026-09-28 02:58:29 on dist-test-slave-k5rg"
server_key: "9981558bc852d17d33dbf8776b92afd9"
server_key_iv: "984c4baf808832974961c78aa90401e6"
server_key_version: "encryptionkey@0"
I20260928 02:58:29.064726 14842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:29.091777 14842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:29.096071 14842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:29.096333 14842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:29.098973 14842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:29.099078 14842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:29.099192 14842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:29.099275 14842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:58:29 dist-test-slave-k5rg krb5kdc[14726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564309, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:29.123929 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:29.099410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:52499 (local address 127.12.182.126:34775)
0928 02:58:29.102013 (+ 2603us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:29.102018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:29.102042 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:58:29.104153 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:29.104159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:29.104256 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:29.104410 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:29.106928 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.107867 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.110118 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.110663 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.110803 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:29.116399 (+ 5596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:29.116404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:29.116408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:29.116442 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:29.119756 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:29.120821 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:29.120828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:29.120831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:29.120992 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:29.121587 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:29.121593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:29.121596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:29.122017 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:29.122175 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:29.123262 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0928 02:58:29.123520 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2509,"thread_start_us":2408,"threads_started":1}
I20260928 02:58:29.125757 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:29.100177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34775 (local address 127.12.182.65:52499)
0928 02:58:29.101135 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:29.101189 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:29.103799 (+ 2610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:29.104686 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:29.104699 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:29.105582 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:29.106571 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:29.106604 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.108056 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.108061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:29.109971 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:29.109987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.111005 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.111016 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:29.111551 (+ 535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:29.112461 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:29.112499 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:29.116140 (+ 3641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:29.119943 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:29.119955 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:29.119980 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:29.120680 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:29.121122 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:29.121129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:29.121133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:29.121443 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:29.122178 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:29.122188 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:29.122764 (+ 576us) client_negotiation.cc:769] Sending connection context
0928 02:58:29.123144 (+ 380us) client_negotiation.cc:240] Negotiation successful
0928 02:58:29.123823 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":377,"threads_started":1}
I20260928 02:58:29.133780 14842 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:39759
I20260928 02:58:29.133862 14974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:39759 every 8 connection(s)
I20260928 02:58:29.134999 14842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:58:29.137835 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 14842
I20260928 02:58:29.138039 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:58:29.138854 13017 external_mini_cluster.cc:1428] Setting key b3ab7fa1e278fb5719f1d25d41b885f3
I20260928 02:58:29.141644 14975 heartbeater.cc:347] Connected to a master server at 127.12.182.126:34775
I20260928 02:58:29.141947 14975 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:29.142961 14975 heartbeater.cc:511] Master 127.12.182.126:34775 requested a full tablet report, sending...
I20260928 02:58:29.145571 14771 ts_manager.cc:194] Registered new tserver with Master: a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759)
I20260928 02:58:29.147730 14771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:52499
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
I20260928 02:58:29.164119 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:29.153238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:50599 (local address 127.12.182.126:34775)
0928 02:58:29.153412 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:29.153418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:29.153668 (+ 250us) server_negotiation.cc:407] Connection header received
0928 02:58:29.153893 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:29.153897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:29.153956 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:29.154034 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:29.154903 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.155570 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.156830 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.157213 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.157315 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:29.159067 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:29.159072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:29.159076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:29.159105 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:29.161427 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:29.162202 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:29.162209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:29.162212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:29.162360 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:29.162742 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:29.162746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:29.162749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:29.163036 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:29.163179 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:29.163573 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 02:58:29.163772 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:58:29.164163 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:29.152980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34775 (local address 127.12.182.65:50599)
0928 02:58:29.153539 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:29.153557 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:29.153763 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:29.154073 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:29.154078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:29.154513 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:29.154767 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:29.154776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.155699 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.155705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:29.156704 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:29.156716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.157390 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.157397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:29.157549 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:29.158128 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:29.158145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:29.158907 (+ 762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:29.161544 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:29.161550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:29.161555 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:29.162051 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:29.162481 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:29.162484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:29.162486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:29.162641 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:29.163147 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:29.163152 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:29.163319 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 02:58:29.163435 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:58:29.163720 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":441,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:58:29.186123 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:34775
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:29.490109 14982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:29.490365 14982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:29.490484 14982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:29.497512 14982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:29.497634 14982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:29.497745 14982 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:58:29.497807 14982 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:58:29.497902 14982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:58:29.507211 14982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.182.126:34775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:29.508373 14982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:29.509655 14982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:29.521840 14987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:29.521888 14990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:29.521806 14988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:29.522397 14982 server_base.cc:1034] running on GCE node
I20260928 02:58:29.522962 14982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:29.523800 14982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:29.525000 14982 hybrid_clock.cc:648] HybridClock initialized: now 1790564309524958 us; error 33 us; skew 500 ppm
I20260928 02:58:29.525290 14982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:29 dist-test-slave-k5rg krb5kdc[14726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564309, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:29.528172 14982 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:58:29.529575 14982 webserver.cc:460] Webserver started at http://127.12.182.66:42879/ using document root <none> and password file <none>
I20260928 02:58:29.530220 14982 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:29.530331 14982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:29.530606 14982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:29.533344 14982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "1631b004512b4e28847b971d97ce563a"
format_stamp: "Formatted at 2026-09-28 02:58:29 on dist-test-slave-k5rg"
server_key: "48f5f7f9607d596bd59cb3fe96e26e38"
server_key_iv: "e22a4704db8f37e4eecaf37b168049d6"
server_key_version: "encryptionkey@0"
I20260928 02:58:29.534233 14982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "1631b004512b4e28847b971d97ce563a"
format_stamp: "Formatted at 2026-09-28 02:58:29 on dist-test-slave-k5rg"
server_key: "48f5f7f9607d596bd59cb3fe96e26e38"
server_key_iv: "e22a4704db8f37e4eecaf37b168049d6"
server_key_version: "encryptionkey@0"
I20260928 02:58:29.541801 14982 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260928 02:58:29.545883 14997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:29.547369 14982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:29.547535 14982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "1631b004512b4e28847b971d97ce563a"
format_stamp: "Formatted at 2026-09-28 02:58:29 on dist-test-slave-k5rg"
server_key: "48f5f7f9607d596bd59cb3fe96e26e38"
server_key_iv: "e22a4704db8f37e4eecaf37b168049d6"
server_key_version: "encryptionkey@0"
I20260928 02:58:29.547756 14982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:29.560894 14982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:29.564975 14982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:29.565204 14982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:29.567687 14982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:29.567775 14982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:29.567878 14982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:29.567950 14982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:58:29 dist-test-slave-k5rg krb5kdc[14726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564309, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:29.594933 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:29.569371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:34765 (local address 127.12.182.126:34775)
0928 02:58:29.569580 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:29.569586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:29.569883 (+ 297us) server_negotiation.cc:407] Connection header received
0928 02:58:29.572557 (+ 2674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:29.572564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:29.572652 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:29.572758 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:29.575030 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.575945 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.578469 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.579219 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.579350 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:29.587908 (+ 8558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:29.587916 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:29.587921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:29.587963 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:29.591266 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:29.592278 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:29.592285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:29.592289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:29.592449 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:29.593039 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:29.593045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:29.593048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:29.593505 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:29.593685 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:29.594370 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 02:58:29.594598 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 02:58:29.595920 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:29.568861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34775 (local address 127.12.182.66:34765)
0928 02:58:29.569711 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:29.569767 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:29.572209 (+ 2442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:29.573042 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:29.573056 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:29.573890 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:29.574819 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:29.574846 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.576161 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.576167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:29.578312 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:29.578332 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.579483 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.579492 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:29.579703 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:29.584720 (+ 5017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:29.584756 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:29.587646 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:29.591412 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:29.591422 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:29.591440 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:29.592108 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:29.592592 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:29.592597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:29.592601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:29.592890 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:29.593668 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:29.593676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:29.594027 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 02:58:29.594253 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 02:58:29.594790 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":342,"threads_started":1}
I20260928 02:58:29.613286 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:29.601376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34775 (local address 127.12.182.66:47065)
0928 02:58:29.601935 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:29.601951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:29.602068 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:29.602340 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:29.602345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:29.602775 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:29.603046 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:29.603055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.604012 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.604018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:29.605032 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:29.605043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.605890 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.605901 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:29.606048 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:29.606623 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:29.606640 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:29.607347 (+ 707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:29.610374 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:29.610379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:29.610384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:29.610858 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:29.611264 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:29.611269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:29.611272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:29.611445 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:29.612091 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:29.612095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:29.612347 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 02:58:29.612502 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:58:29.612826 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":311,"threads_started":1}
I20260928 02:58:29.613981 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:29.601287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:47065 (local address 127.12.182.126:34775)
0928 02:58:29.601439 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:29.601443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:29.602026 (+ 583us) server_negotiation.cc:407] Connection header received
0928 02:58:29.602176 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:29.602179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:29.602234 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:29.602305 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:29.603173 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.603891 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.605159 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:29.605708 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:29.605815 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:29.607525 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:29.607529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:29.607532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:29.607561 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:29.610253 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:29.610977 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:29.610983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:29.610987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:29.611151 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:29.611602 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:29.611606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:29.611608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:29.611993 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:29.612130 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:29.613567 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0928 02:58:29.613775 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:58:29.623176 14982 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:38673
I20260928 02:58:29.623229 15116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:38673 every 8 connection(s)
I20260928 02:58:29.624280 14982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:58:29.626118 15117 heartbeater.cc:347] Connected to a master server at 127.12.182.126:34775
I20260928 02:58:29.626322 15117 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:29.626953 15117 heartbeater.cc:511] Master 127.12.182.126:34775 requested a full tablet report, sending...
I20260928 02:58:29.628113 14771 ts_manager.cc:194] Registered new tserver with Master: 1631b004512b4e28847b971d97ce563a (127.12.182.66:38673)
I20260928 02:58:29.628954 14771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:34765
I20260928 02:58:29.631418 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 14982
I20260928 02:58:29.631564 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:58:29.631982 13017 external_mini_cluster.cc:1428] Setting key 62dfddd34a577341ffb699d4bcc84412
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:58:29.666416 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:34775
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:29.980345 15121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:29.980594 15121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:29.980701 15121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:29.988047 15121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:29.988154 15121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:29.988245 15121 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:58:29.988297 15121 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:58:29.988358 15121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:58:29.998116 15121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.182.126:34775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:29.999356 15121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:30.000578 15121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:30.012569 15126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:30.012598 15129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:30.012537 15127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:30.013202 15121 server_base.cc:1034] running on GCE node
I20260928 02:58:30.013804 15121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:30.014674 15121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:30.015875 15121 hybrid_clock.cc:648] HybridClock initialized: now 1790564310015827 us; error 40 us; skew 500 ppm
I20260928 02:58:30.016201 15121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:30 dist-test-slave-k5rg krb5kdc[14726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564310, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:30.019256 15121 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:58:30.020638 15121 webserver.cc:460] Webserver started at http://127.12.182.67:44897/ using document root <none> and password file <none>
I20260928 02:58:30.021265 15121 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:30.021375 15121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:30.021699 15121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:30.024596 15121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "94d4e1bf13e24b6294d56c5ac33972c9"
format_stamp: "Formatted at 2026-09-28 02:58:30 on dist-test-slave-k5rg"
server_key: "6302bd302c0c5ecc46061dcfebbff014"
server_key_iv: "f638780b581f40e3d4f7e499ff74e92b"
server_key_version: "encryptionkey@0"
I20260928 02:58:30.025326 15121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "94d4e1bf13e24b6294d56c5ac33972c9"
format_stamp: "Formatted at 2026-09-28 02:58:30 on dist-test-slave-k5rg"
server_key: "6302bd302c0c5ecc46061dcfebbff014"
server_key_iv: "f638780b581f40e3d4f7e499ff74e92b"
server_key_version: "encryptionkey@0"
I20260928 02:58:30.030236 15121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 02:58:30.033499 15136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:30.034790 15121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:30.034961 15121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "94d4e1bf13e24b6294d56c5ac33972c9"
format_stamp: "Formatted at 2026-09-28 02:58:30 on dist-test-slave-k5rg"
server_key: "6302bd302c0c5ecc46061dcfebbff014"
server_key_iv: "f638780b581f40e3d4f7e499ff74e92b"
server_key_version: "encryptionkey@0"
I20260928 02:58:30.035125 15121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:30.049423 15121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:30.053172 15121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:30.053413 15121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:30.056074 15121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:30.056205 15121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:30.056354 15121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:30.056464 15121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:58:30 dist-test-slave-k5rg krb5kdc[14726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564310, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:30.078580 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.056393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:57919 (local address 127.12.182.126:34775)
0928 02:58:30.056562 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.056568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.057860 (+ 1292us) server_negotiation.cc:407] Connection header received
0928 02:58:30.059883 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.059889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.059968 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.060054 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:30.062127 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.063104 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.066064 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.066471 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.066588 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.071227 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:30.071233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:30.071238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:30.071281 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:30.074447 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:30.075675 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:30.075681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:30.075685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:30.075846 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:30.076412 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:30.076417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:30.076421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:30.076811 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:30.076954 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.077959 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.078237 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:58:30.079643 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.056961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34775 (local address 127.12.182.67:57919)
0928 02:58:30.057725 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.057763 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.059544 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.060289 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.060296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.060895 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:30.061958 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.061983 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.063276 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.063283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.065898 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.065919 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.066718 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.066726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.067032 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.067787 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:30.067823 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:30.070996 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:30.074637 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:30.074646 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:30.074663 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:30.075507 (+ 844us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:30.075943 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:30.075952 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:30.075956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:30.076245 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:30.077025 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:30.077031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:30.077520 (+ 489us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.077832 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.078353 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":324,"threads_started":1}
I20260928 02:58:30.099427 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.085075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34775 (local address 127.12.182.67:35125)
0928 02:58:30.085722 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.085745 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.085965 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.086289 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.086294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.086945 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:30.087350 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.087363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.088310 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.088315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.090164 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.090183 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.091038 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.091048 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.091242 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.091939 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:30.091966 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:30.093181 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:30.096576 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:30.096582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:30.096586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:30.097023 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:30.097366 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:30.097371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:30.097375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:30.097523 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:30.098059 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:30.098064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:30.098358 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.098524 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.098917 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":335,"threads_started":1}
I20260928 02:58:30.099495 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.085354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:35125 (local address 127.12.182.126:34775)
0928 02:58:30.085499 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.085512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.085828 (+ 316us) server_negotiation.cc:407] Connection header received
0928 02:58:30.086096 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.086100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.086171 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.086248 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:30.087493 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.088196 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.090312 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.090849 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.090974 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.093398 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:30.093403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:30.093407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:30.093439 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:30.096447 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:30.097128 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:30.097134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:30.097138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:30.097266 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:30.097652 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:30.097657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:30.097660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:30.097960 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:30.098103 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.098958 (+ 855us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.099185 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:58:30.108572 15121 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:33351
I20260928 02:58:30.108618 15255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:33351 every 8 connection(s)
I20260928 02:58:30.109772 15121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:58:30.110857 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 15121
I20260928 02:58:30.111039 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:58:30.111502 13017 external_mini_cluster.cc:1428] Setting key 4928971a062674e66c2c37e5c195da3e
I20260928 02:58:30.111680 15256 heartbeater.cc:347] Connected to a master server at 127.12.182.126:34775
I20260928 02:58:30.112346 15256 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:30.113116 15256 heartbeater.cc:511] Master 127.12.182.126:34775 requested a full tablet report, sending...
I20260928 02:58:30.114687 14771 ts_manager.cc:194] Registered new tserver with Master: 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351)
I20260928 02:58:30.115444 14771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:57919
I20260928 02:58:30.126771 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:58:30 dist-test-slave-k5rg krb5kdc[14726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260928 02:58:30.151687 14975 heartbeater.cc:503] Master 127.12.182.126:34775 was elected leader, sending a full tablet report...
Sep 28 02:58:30 dist-test-slave-k5rg krb5kdc[14726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:30.169793 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.156799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.12.182.126:34775)
0928 02:58:30.156988 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.156993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.157305 (+ 312us) server_negotiation.cc:407] Connection header received
0928 02:58:30.157467 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.157472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.157534 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.157635 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:30.158895 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.159626 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.160781 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.161198 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.161301 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.164930 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:30.164935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:30.164939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:30.164969 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:30.167367 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:30.167930 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:30.167935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:30.167939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:30.168065 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:30.168399 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:30.168404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:30.168408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:30.168684 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:30.168903 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.169190 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.169394 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:58:30.173529 14771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:58:30.180251 14771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:30.203490 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.191733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.12.182.65:39759)
0928 02:58:30.192291 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.192301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.193578 (+ 1277us) server_negotiation.cc:407] Connection header received
0928 02:58:30.194043 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.194052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.194492 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.194666 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:30.195700 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.197362 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.200337 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.201964 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.202079 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.202301 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.202999 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.203185 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":277,"threads_started":1}
I20260928 02:58:30.203624 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.191500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39759 (local address 127.0.0.1:60046)
0928 02:58:30.193406 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.193456 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.193774 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.194987 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.194995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.195070 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:30.195475 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.195487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.197659 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.197665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.200142 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.200159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.202491 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.202498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.202731 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.202784 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.202939 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.203151 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1631,"spinlock_wait_cycles":26240,"thread_start_us":1463,"threads_started":1}
I20260928 02:58:30.207947 14908 tablet_service.cc:1467] Processing CreateTablet for tablet dc7bedf8d5de4c8093f9ec54b0a0790c (DEFAULT_TABLE table=test-table [id=4cef1bc1d0664218b4e5bb80928d91a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:30.209450 14908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7bedf8d5de4c8093f9ec54b0a0790c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:30.210999 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.195516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:33351 (local address 127.0.0.1:43920)
0928 02:58:30.203244 (+ 7728us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.203261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.203592 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.204467 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.204471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.204495 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:30.204847 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.204860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.206694 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.206699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.208463 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.208474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.210291 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.210296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.210440 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.210471 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.210567 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.210685 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7618,"thread_start_us":7855,"threads_started":1}
I20260928 02:58:30.211097 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.195277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.12.182.67:33351)
0928 02:58:30.195934 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.195943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.203363 (+ 7420us) server_negotiation.cc:407] Connection header received
0928 02:58:30.203810 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.203817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.204013 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.204170 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:30.205047 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.206567 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.208594 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.209927 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.210039 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.210223 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.210629 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.210788 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":354,"threads_started":1}
I20260928 02:58:30.212349 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.191912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:38673 (local address 127.0.0.1:50832)
0928 02:58:30.194923 (+ 3011us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.194938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.195071 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.206057 (+ 10986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.206062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.206085 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:30.206374 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.206382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.208066 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.208071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.210061 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.210076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.211632 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.211638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.211801 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.211831 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.211906 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.212025 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2886,"spinlock_wait_cycles":3200,"thread_start_us":2769,"threads_started":1}
I20260928 02:58:30.213063 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.192437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.12.182.66:38673)
0928 02:58:30.205584 (+ 13147us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.205590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.205635 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:58:30.205712 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.205718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.205875 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.206034 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:30.206529 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.207918 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.210230 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.211402 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.212554 (+ 1152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.212699 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.212747 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.212855 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13008,"thread_start_us":12875,"threads_started":1}
I20260928 02:58:30.214821 15186 tablet_service.cc:1467] Processing CreateTablet for tablet dc7bedf8d5de4c8093f9ec54b0a0790c (DEFAULT_TABLE table=test-table [id=4cef1bc1d0664218b4e5bb80928d91a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:30.216259 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7bedf8d5de4c8093f9ec54b0a0790c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:30.216504 15047 tablet_service.cc:1467] Processing CreateTablet for tablet dc7bedf8d5de4c8093f9ec54b0a0790c (DEFAULT_TABLE table=test-table [id=4cef1bc1d0664218b4e5bb80928d91a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:30.217573 15047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7bedf8d5de4c8093f9ec54b0a0790c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:30.223456 15275 tablet_bootstrap.cc:492] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: Bootstrap starting.
I20260928 02:58:30.226884 15275 tablet_bootstrap.cc:654] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:30.228118 15275 log.cc:824] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:30.228633 15276 tablet_bootstrap.cc:492] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: Bootstrap starting.
I20260928 02:58:30.229912 15278 tablet_bootstrap.cc:492] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: Bootstrap starting.
I20260928 02:58:30.231609 15275 tablet_bootstrap.cc:492] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: No bootstrap required, opened a new log
I20260928 02:58:30.231868 15275 ts_tablet_manager.cc:1397] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:58:30.232363 15276 tablet_bootstrap.cc:654] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:30.232774 15278 tablet_bootstrap.cc:654] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:30.233489 15276 log.cc:824] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:30.233645 15278 log.cc:824] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:30.236200 15276 tablet_bootstrap.cc:492] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: No bootstrap required, opened a new log
I20260928 02:58:30.236331 15278 tablet_bootstrap.cc:492] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: No bootstrap required, opened a new log
I20260928 02:58:30.236461 15276 ts_tablet_manager.cc:1397] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 02:58:30.236544 15278 ts_tablet_manager.cc:1397] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 02:58:30.239706 15275 raft_consensus.cc:348] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.240034 15275 raft_consensus.cc:374] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:30.240125 15275 raft_consensus.cc:729] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a950b7016e0d49ae874677d2ffb09265, State: Initialized, Role: FOLLOWER
I20260928 02:58:30.240654 15275 consensus_queue.cc:260] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.246779 15275 ts_tablet_manager.cc:1428] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260928 02:58:30.246531 15278 raft_consensus.cc:348] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.246922 15278 raft_consensus.cc:374] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:30.247058 15278 raft_consensus.cc:729] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1631b004512b4e28847b971d97ce563a, State: Initialized, Role: FOLLOWER
I20260928 02:58:30.247301 15276 raft_consensus.cc:348] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.247692 15276 raft_consensus.cc:374] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:30.247618 15278 consensus_queue.cc:260] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.247792 15276 raft_consensus.cc:729] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4e1bf13e24b6294d56c5ac33972c9, State: Initialized, Role: FOLLOWER
I20260928 02:58:30.248328 15276 consensus_queue.cc:260] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.248720 15117 heartbeater.cc:503] Master 127.12.182.126:34775 was elected leader, sending a full tablet report...
I20260928 02:58:30.249275 15278 ts_tablet_manager.cc:1428] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260928 02:58:30.251041 15276 ts_tablet_manager.cc:1428] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260928 02:58:30.251060 15256 heartbeater.cc:503] Master 127.12.182.126:34775 was elected leader, sending a full tablet report...
I20260928 02:58:30.333684 15283 raft_consensus.cc:482] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:30.333974 15283 raft_consensus.cc:504] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.335572 15283 leader_election.cc:288] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759), 1631b004512b4e28847b971d97ce563a (127.12.182.66:38673)
I20260928 02:58:30.341012 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.336002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39759 (local address 127.12.182.67:55215)
0928 02:58:30.336195 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.336210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.336343 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.336644 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.336648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.336675 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:30.336889 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.336900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.337930 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.337934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.339227 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.339241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.340315 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.340321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.340508 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.340531 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.340609 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.340740 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:58:30.341012 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.335917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:55215 (local address 127.12.182.65:39759)
0928 02:58:30.336109 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.336114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.336275 (+ 161us) server_negotiation.cc:407] Connection header received
0928 02:58:30.336477 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.336481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.336539 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.336611 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:30.337029 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.337785 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.339391 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.340175 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.340317 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.340401 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.340648 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.340771 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:58:30.342059 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.336428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:38673 (local address 127.12.182.67:56551)
0928 02:58:30.337113 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.337127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.337239 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.337532 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.337536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.337560 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:30.337856 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.337865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.338921 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.338926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.340315 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.340327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.341442 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.341447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.341642 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.341678 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.341752 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.341837 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":379,"threads_started":1}
I20260928 02:58:30.342163 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.336043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:56551 (local address 127.12.182.66:38673)
0928 02:58:30.336232 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.336236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.337215 (+ 979us) server_negotiation.cc:407] Connection header received
0928 02:58:30.337360 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.337364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.337429 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.337498 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:30.338018 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.338793 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.340459 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.341259 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.341415 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.341483 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.341799 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.341929 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:58:30.342002 14928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7bedf8d5de4c8093f9ec54b0a0790c" candidate_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a950b7016e0d49ae874677d2ffb09265" is_pre_election: true
I20260928 02:58:30.342579 14928 raft_consensus.cc:2454] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d4e1bf13e24b6294d56c5ac33972c9 in term 0.
I20260928 02:58:30.343173 15067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7bedf8d5de4c8093f9ec54b0a0790c" candidate_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1631b004512b4e28847b971d97ce563a" is_pre_election: true
I20260928 02:58:30.343362 15139 leader_election.cc:302] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94d4e1bf13e24b6294d56c5ac33972c9, a950b7016e0d49ae874677d2ffb09265; no voters:
I20260928 02:58:30.343676 15067 raft_consensus.cc:2454] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d4e1bf13e24b6294d56c5ac33972c9 in term 0.
I20260928 02:58:30.343732 15283 raft_consensus.cc:2781] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:30.343870 15283 raft_consensus.cc:482] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:30.343940 15283 raft_consensus.cc:3037] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:30.346066 15283 raft_consensus.cc:504] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.346616 15283 leader_election.cc:288] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [CANDIDATE]: Term 1 election: Requested vote from peers a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759), 1631b004512b4e28847b971d97ce563a (127.12.182.66:38673)
I20260928 02:58:30.347059 15067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7bedf8d5de4c8093f9ec54b0a0790c" candidate_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1631b004512b4e28847b971d97ce563a"
I20260928 02:58:30.347059 14928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7bedf8d5de4c8093f9ec54b0a0790c" candidate_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a950b7016e0d49ae874677d2ffb09265"
I20260928 02:58:30.347254 15067 raft_consensus.cc:3037] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:30.347263 14928 raft_consensus.cc:3037] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:30.350283 14928 raft_consensus.cc:2454] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d4e1bf13e24b6294d56c5ac33972c9 in term 1.
I20260928 02:58:30.350283 15067 raft_consensus.cc:2454] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d4e1bf13e24b6294d56c5ac33972c9 in term 1.
I20260928 02:58:30.350672 15140 leader_election.cc:302] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1631b004512b4e28847b971d97ce563a, 94d4e1bf13e24b6294d56c5ac33972c9, a950b7016e0d49ae874677d2ffb09265; no voters:
I20260928 02:58:30.350978 15283 raft_consensus.cc:2781] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:30.351368 15283 raft_consensus.cc:686] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 LEADER]: Becoming Leader. State: Replica: 94d4e1bf13e24b6294d56c5ac33972c9, State: Running, Role: LEADER
I20260928 02:58:30.351779 15283 consensus_queue.cc:237] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.356127 14771 catalog_manager.cc:5495] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67). New cstate: current_term: 1 leader_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } health_report { overall_health: HEALTHY } } }
W20260928 02:58:30.361142 15257 tablet.cc:2369] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:58:30.375985 15118 tablet.cc:2369] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:58:30.413178 14976 tablet.cc:2369] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:58:30 dist-test-slave-k5rg krb5kdc[14726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564308, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:30.419879 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.403718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34775 (local address 127.0.0.1:48078)
0928 02:58:30.404259 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.404274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.404390 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.404737 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.404740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.405480 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:30.405845 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.405859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.406775 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.406779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.407728 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.407739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.408460 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.408465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.408582 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.413139 (+ 4557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:30.413156 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:30.415221 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:30.417763 (+ 2542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:30.417773 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:30.417779 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:30.418173 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:30.418492 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:30.418495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:30.418497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:30.418621 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:30.419059 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:30.419064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:30.419211 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.419339 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.419551 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":297,"threads_started":1}
I20260928 02:58:30.419910 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.403648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.12.182.126:34775)
0928 02:58:30.403823 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.403829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.404348 (+ 519us) server_negotiation.cc:407] Connection header received
0928 02:58:30.404493 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.404496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.404571 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.404646 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:30.405967 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.406664 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.407859 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.408308 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.408396 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.415393 (+ 6997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:30.415398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:30.415401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:30.415429 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:30.417636 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:30.418270 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:30.418273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:30.418276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:30.418389 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:30.418701 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:30.418704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:30.418706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:30.418959 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:30.419065 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.419421 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.419654 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":56}
I20260928 02:58:30.424310 14771 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:48078:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:58:30.425212 14771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:30.431329 14908 tablet_service.cc:1467] Processing CreateTablet for tablet b2640ab67f3648b3ab6f6856a95df9ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa381cb39b4f411787b9e2e3d008b7bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:58:30.431628 15186 tablet_service.cc:1467] Processing CreateTablet for tablet b2640ab67f3648b3ab6f6856a95df9ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa381cb39b4f411787b9e2e3d008b7bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:58:30.431628 15047 tablet_service.cc:1467] Processing CreateTablet for tablet b2640ab67f3648b3ab6f6856a95df9ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa381cb39b4f411787b9e2e3d008b7bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:58:30.432310 14908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2640ab67f3648b3ab6f6856a95df9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:30.432468 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2640ab67f3648b3ab6f6856a95df9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:30.432468 15047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2640ab67f3648b3ab6f6856a95df9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:30.437830 15275 tablet_bootstrap.cc:492] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: Bootstrap starting.
I20260928 02:58:30.439221 15275 tablet_bootstrap.cc:654] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:30.439250 15276 tablet_bootstrap.cc:492] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: Bootstrap starting.
I20260928 02:58:30.440723 15278 tablet_bootstrap.cc:492] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: Bootstrap starting.
I20260928 02:58:30.440753 15276 tablet_bootstrap.cc:654] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:30.441510 15275 tablet_bootstrap.cc:492] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: No bootstrap required, opened a new log
I20260928 02:58:30.441663 15275 ts_tablet_manager.cc:1397] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 02:58:30.442344 15278 tablet_bootstrap.cc:654] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:30.442257 15275 raft_consensus.cc:348] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.442442 15275 raft_consensus.cc:374] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:30.442544 15275 raft_consensus.cc:729] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a950b7016e0d49ae874677d2ffb09265, State: Initialized, Role: FOLLOWER
I20260928 02:58:30.442771 15275 consensus_queue.cc:260] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.443346 15275 ts_tablet_manager.cc:1428] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:58:30.443257 15281 tablet_replica.cc:439] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } } }
I20260928 02:58:30.443668 15281 tablet_replica.cc:442] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:30.445091 15278 tablet_bootstrap.cc:492] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: No bootstrap required, opened a new log
I20260928 02:58:30.445181 15276 tablet_bootstrap.cc:492] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: No bootstrap required, opened a new log
I20260928 02:58:30.445225 15278 ts_tablet_manager.cc:1397] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:58:30.445308 15276 ts_tablet_manager.cc:1397] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260928 02:58:30.445958 15278 raft_consensus.cc:348] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.446075 15276 raft_consensus.cc:348] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.446190 15278 raft_consensus.cc:374] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:30.446269 15276 raft_consensus.cc:374] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:30.446282 15278 raft_consensus.cc:729] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1631b004512b4e28847b971d97ce563a, State: Initialized, Role: FOLLOWER
I20260928 02:58:30.446360 15276 raft_consensus.cc:729] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4e1bf13e24b6294d56c5ac33972c9, State: Initialized, Role: FOLLOWER
I20260928 02:58:30.446550 15276 consensus_queue.cc:260] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.446511 15278 consensus_queue.cc:260] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.446894 15283 tablet_replica.cc:439] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } } }
I20260928 02:58:30.447054 15276 ts_tablet_manager.cc:1428] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:58:30.447100 15278 ts_tablet_manager.cc:1428] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:58:30.447154 15283 tablet_replica.cc:442] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:30.447124 15282 tablet_replica.cc:439] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } } }
I20260928 02:58:30.448015 15282 tablet_replica.cc:442] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:30.663529 15282 raft_consensus.cc:482] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:30.663756 15282 raft_consensus.cc:504] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.665359 15282 leader_election.cc:288] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759), 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351)
I20260928 02:58:30.672603 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.666000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:50189 (local address 127.12.182.67:33351)
0928 02:58:30.666162 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.666167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.667000 (+ 833us) server_negotiation.cc:407] Connection header received
0928 02:58:30.667124 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.667130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.667205 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.667296 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:30.667832 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.668603 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.670934 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.671665 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.671788 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.671858 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.672218 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.672349 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:58:30.672637 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.665864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:37583 (local address 127.12.182.65:39759)
0928 02:58:30.666058 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.666064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.666430 (+ 366us) server_negotiation.cc:407] Connection header received
0928 02:58:30.666621 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.666626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.666687 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.666759 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:30.667387 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.668311 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.670884 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.671704 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.671873 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.671954 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.672269 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.672391 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:58:30.672704 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.666281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:33351 (local address 127.12.182.66:50189)
0928 02:58:30.666898 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.666912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.667021 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.667319 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.667323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.667346 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:30.667680 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.667691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.668714 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.668720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.670734 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.670747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.671869 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.671880 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.672072 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.672097 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.672167 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.672279 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":34,"thread_start_us":319,"threads_started":1}
I20260928 02:58:30.672747 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.665561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39759 (local address 127.12.182.66:37583)
0928 02:58:30.666326 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.666344 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.666489 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.666792 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.666797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.666824 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:30.667215 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.667227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.668446 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.668450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.670732 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.670747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.671941 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.671945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.672117 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.672140 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.672228 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.672348 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":64,"thread_start_us":463,"threads_started":1}
I20260928 02:58:30.673296 14928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2640ab67f3648b3ab6f6856a95df9ea" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a950b7016e0d49ae874677d2ffb09265" is_pre_election: true
I20260928 02:58:30.673522 14928 raft_consensus.cc:2454] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 0.
I20260928 02:58:30.673519 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2640ab67f3648b3ab6f6856a95df9ea" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" is_pre_election: true
I20260928 02:58:30.673902 15206 raft_consensus.cc:2454] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 0.
I20260928 02:58:30.673969 15000 leader_election.cc:302] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1631b004512b4e28847b971d97ce563a, a950b7016e0d49ae874677d2ffb09265; no voters:
I20260928 02:58:30.674335 15282 raft_consensus.cc:2781] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:30.674444 15282 raft_consensus.cc:482] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:30.674551 15282 raft_consensus.cc:3037] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:30.676640 15282 raft_consensus.cc:504] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.677196 15282 leader_election.cc:288] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 election: Requested vote from peers a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759), 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351)
I20260928 02:58:30.677631 14928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2640ab67f3648b3ab6f6856a95df9ea" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a950b7016e0d49ae874677d2ffb09265"
I20260928 02:58:30.677703 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2640ab67f3648b3ab6f6856a95df9ea" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4e1bf13e24b6294d56c5ac33972c9"
I20260928 02:58:30.677866 14928 raft_consensus.cc:3037] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:30.677882 15206 raft_consensus.cc:3037] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:30.680727 14928 raft_consensus.cc:2454] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 1.
I20260928 02:58:30.680742 15206 raft_consensus.cc:2454] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 1.
I20260928 02:58:30.681126 15001 leader_election.cc:302] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1631b004512b4e28847b971d97ce563a, 94d4e1bf13e24b6294d56c5ac33972c9; no voters:
I20260928 02:58:30.681420 15282 raft_consensus.cc:2781] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:30.681851 15282 raft_consensus.cc:686] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [term 1 LEADER]: Becoming Leader. State: Replica: 1631b004512b4e28847b971d97ce563a, State: Running, Role: LEADER
I20260928 02:58:30.682332 15282 consensus_queue.cc:237] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.685645 15298 tablet_replica.cc:439] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: TxnStatusTablet state changed. Reason: New leader 1631b004512b4e28847b971d97ce563a. Latest consensus state: current_term: 1 leader_uuid: "1631b004512b4e28847b971d97ce563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } } }
I20260928 02:58:30.686029 15298 tablet_replica.cc:442] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:58:30.686650 15300 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:58:30.686869 15300 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:58:30.687124 14771 catalog_manager.cc:5495] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1631b004512b4e28847b971d97ce563a (127.12.182.66). New cstate: current_term: 1 leader_uuid: "1631b004512b4e28847b971d97ce563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } health_report { overall_health: UNKNOWN } } }
I20260928 02:58:30.735975 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.729862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:38673 (local address 127.0.0.1:50836)
0928 02:58:30.730075 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.730091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.730216 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.730727 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.730731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.730740 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:58:30.730922 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.730929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.732367 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.732370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.733331 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.733341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.734322 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.734327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.734468 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.734557 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:58:30.735563 (+ 1006us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:58:30.735581 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.735639 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.735751 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:58:30.736069 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.729928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.12.182.66:38673)
0928 02:58:30.730110 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.730117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.730137 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:58:30.730460 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.730465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.730573 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.730725 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:30.731042 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.732260 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.733467 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.734192 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.734321 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.734753 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:30.735439 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:30.735570 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.735665 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.735749 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:58:30.743062 14928 raft_consensus.cc:1261] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Refusing update from remote peer 1631b004512b4e28847b971d97ce563a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:30.743054 15206 raft_consensus.cc:1261] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Refusing update from remote peer 1631b004512b4e28847b971d97ce563a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:30.744005 15298 consensus_queue.cc:1035] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:30.744371 15282 consensus_queue.cc:1035] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:30.751158 15283 tablet_replica.cc:439] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: TxnStatusTablet state changed. Reason: New leader 1631b004512b4e28847b971d97ce563a. Latest consensus state: current_term: 1 leader_uuid: "1631b004512b4e28847b971d97ce563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } } }
I20260928 02:58:30.751499 15283 tablet_replica.cc:442] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:30.755110 15281 tablet_replica.cc:439] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: TxnStatusTablet state changed. Reason: New leader 1631b004512b4e28847b971d97ce563a. Latest consensus state: current_term: 1 leader_uuid: "1631b004512b4e28847b971d97ce563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } } }
I20260928 02:58:30.755388 15281 tablet_replica.cc:442] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:30.756162 15303 mvcc.cc:205] Tried to move back new op lower bound from 7334151416794722304 to 7334151416558436352. Current Snapshot: MvccSnapshot[applied={T|T < 7334151416794722304}]
I20260928 02:58:30.756742 15298 tablet_replica.cc:439] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1631b004512b4e28847b971d97ce563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } } }
I20260928 02:58:30.756989 15298 tablet_replica.cc:442] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:58:30.757555 15298 tablet_replica.cc:439] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1631b004512b4e28847b971d97ce563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } } }
I20260928 02:58:30.757813 15298 tablet_replica.cc:442] T b2640ab67f3648b3ab6f6856a95df9ea P 1631b004512b4e28847b971d97ce563a: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:58:30.772230 15283 tablet_replica.cc:439] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1631b004512b4e28847b971d97ce563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } } }
I20260928 02:58:30.772553 15283 tablet_replica.cc:442] T b2640ab67f3648b3ab6f6856a95df9ea P 94d4e1bf13e24b6294d56c5ac33972c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:30.777434 15281 tablet_replica.cc:439] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1631b004512b4e28847b971d97ce563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } } }
I20260928 02:58:30.777733 15281 tablet_replica.cc:442] T b2640ab67f3648b3ab6f6856a95df9ea P a950b7016e0d49ae874677d2ffb09265: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:58:30.784047 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.775871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.12.182.67:33351)
0928 02:58:30.776052 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.776058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.776592 (+ 534us) server_negotiation.cc:407] Connection header received
0928 02:58:30.776817 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.776822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.776926 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.777060 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:30.778987 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.780016 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.782069 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.782465 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.782553 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.783024 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:30.783503 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:30.783603 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.783712 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.783775 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:58:30.801348 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.795470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:51549 (local address 127.12.182.66:38673)
0928 02:58:30.795750 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.795755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.796105 (+ 350us) server_negotiation.cc:407] Connection header received
0928 02:58:30.796298 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.796302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.796372 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.796443 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:30.796946 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.797773 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.799735 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.800168 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.800258 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.800782 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:30.800880 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:30.800939 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.801080 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.801139 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:58:30.801585 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.795396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:38673 (local address 127.12.182.67:51549)
0928 02:58:30.796001 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.796024 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.796178 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.796474 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.796479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.796492 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:58:30.796816 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.796827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.797923 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.797929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.799610 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.799625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.800326 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.800332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.800516 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.800584 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:58:30.800963 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:58:30.800980 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.801056 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.801195 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":298,"threads_started":1}
I20260928 02:58:30.811095 14928 raft_consensus.cc:1261] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Refusing update from remote peer 94d4e1bf13e24b6294d56c5ac33972c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:30.811445 15067 raft_consensus.cc:1261] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Refusing update from remote peer 94d4e1bf13e24b6294d56c5ac33972c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:30.811650 15285 consensus_queue.cc:1035] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:30.812103 15283 consensus_queue.cc:1035] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:30.816789 15304 mvcc.cc:205] Tried to move back new op lower bound from 7334151417077170176 to 7334151415204290560. Current Snapshot: MvccSnapshot[applied={T|T < 7334151417077170176}]
I20260928 02:58:30.848466 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.841510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:55135 (local address 127.12.182.67:33351)
0928 02:58:30.841702 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:30.841709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:30.842438 (+ 729us) server_negotiation.cc:407] Connection header received
0928 02:58:30.842660 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:30.842665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:30.842739 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:30.842807 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:30.843495 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.844332 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.846626 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.847011 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.847109 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.847605 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:30.847730 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:30.847816 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:30.848136 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 02:58:30.848210 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:58:30.848892 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:30.841180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:33351 (local address 127.12.182.66:55135)
0928 02:58:30.842333 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:30.842351 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:30.842516 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:30.842845 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:30.842850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:30.842863 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:58:30.843271 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.843283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.844461 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.844467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.846495 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:30.846511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:30.847181 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:30.847188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:30.847378 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:30.847466 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:58:30.848004 (+ 538us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:58:30.848016 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:58:30.848096 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:58:30.848237 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":806,"threads_started":1}
I20260928 02:58:30.882006 14770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:30.882380 14770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:30.886353 14770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a: Sending DeleteTablet for 3 replicas of tablet dc7bedf8d5de4c8093f9ec54b0a0790c
I20260928 02:58:30.887380 14908 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7bedf8d5de4c8093f9ec54b0a0790c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:30 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:60046
I20260928 02:58:30.887544 15047 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7bedf8d5de4c8093f9ec54b0a0790c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:30 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:50832
I20260928 02:58:30.888711 15186 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7bedf8d5de4c8093f9ec54b0a0790c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:30 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:43920
I20260928 02:58:30.895499 14770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:58:30.896353 14770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:30.907420 14907 tablet_service.cc:1467] Processing CreateTablet for tablet ea3978c7e80842e9a4acaf83e614e79f (DEFAULT_TABLE table=test-table [id=f0f05f6ddee14f6bb22f554c6b74fdf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:30.907797 15185 tablet_service.cc:1467] Processing CreateTablet for tablet ea3978c7e80842e9a4acaf83e614e79f (DEFAULT_TABLE table=test-table [id=f0f05f6ddee14f6bb22f554c6b74fdf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:30.908021 14907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3978c7e80842e9a4acaf83e614e79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:30.908291 15185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3978c7e80842e9a4acaf83e614e79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:30.914531 15046 tablet_service.cc:1467] Processing CreateTablet for tablet ea3978c7e80842e9a4acaf83e614e79f (DEFAULT_TABLE table=test-table [id=f0f05f6ddee14f6bb22f554c6b74fdf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:30.914860 15275 tablet_bootstrap.cc:492] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265: Bootstrap starting.
I20260928 02:58:30.915072 15046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3978c7e80842e9a4acaf83e614e79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:30.916474 15275 tablet_bootstrap.cc:654] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:30.918503 15276 tablet_bootstrap.cc:492] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9: Bootstrap starting.
I20260928 02:58:30.920070 15276 tablet_bootstrap.cc:654] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:30.921758 15278 tablet_bootstrap.cc:492] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a: Bootstrap starting.
I20260928 02:58:30.923487 15278 tablet_bootstrap.cc:654] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:30.940593 15278 tablet_bootstrap.cc:492] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a: No bootstrap required, opened a new log
I20260928 02:58:30.940771 15278 ts_tablet_manager.cc:1397] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.007s
I20260928 02:58:30.941637 15278 raft_consensus.cc:348] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.941911 15278 raft_consensus.cc:374] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:30.941987 15278 raft_consensus.cc:729] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1631b004512b4e28847b971d97ce563a, State: Initialized, Role: FOLLOWER
I20260928 02:58:30.942004 15276 tablet_bootstrap.cc:492] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9: No bootstrap required, opened a new log
I20260928 02:58:30.942226 15276 ts_tablet_manager.cc:1397] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.006s
I20260928 02:58:30.942276 15278 consensus_queue.cc:260] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.942860 15278 ts_tablet_manager.cc:1428] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:58:30.943126 15276 raft_consensus.cc:348] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.943403 15276 raft_consensus.cc:374] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:30.943567 15276 raft_consensus.cc:729] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4e1bf13e24b6294d56c5ac33972c9, State: Initialized, Role: FOLLOWER
I20260928 02:58:30.943893 15276 consensus_queue.cc:260] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.946408 15276 ts_tablet_manager.cc:1428] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260928 02:58:30.947160 15326 tablet_replica.cc:330] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: stopping tablet replica
I20260928 02:58:30.947757 15326 raft_consensus.cc:2229] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:58:30.948438 15326 raft_consensus.cc:2258] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:30.948971 15325 tablet_replica.cc:330] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: stopping tablet replica
I20260928 02:58:30.949415 15325 raft_consensus.cc:2229] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:30.950030 15325 raft_consensus.cc:2258] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:30.951400 15275 tablet_bootstrap.cc:492] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265: No bootstrap required, opened a new log
I20260928 02:58:30.951591 15275 ts_tablet_manager.cc:1397] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.019s
I20260928 02:58:30.951925 15326 ts_tablet_manager.cc:1905] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:30.952349 15275 raft_consensus.cc:348] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.952502 15325 ts_tablet_manager.cc:1905] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:30.952596 15275 raft_consensus.cc:374] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:30.952672 15275 raft_consensus.cc:729] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a950b7016e0d49ae874677d2ffb09265, State: Initialized, Role: FOLLOWER
I20260928 02:58:30.952970 15275 consensus_queue.cc:260] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.954236 15275 ts_tablet_manager.cc:1428] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 02:58:30.959877 15325 ts_tablet_manager.cc:1918] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:58:30.960371 15324 tablet_replica.cc:330] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: stopping tablet replica
I20260928 02:58:30.960794 15324 raft_consensus.cc:2229] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:30.961187 15324 raft_consensus.cc:2258] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:30.962922 15324 ts_tablet_manager.cc:1905] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:30.964638 15326 ts_tablet_manager.cc:1918] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:58:30.964802 15326 log.cc:1196] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/dc7bedf8d5de4c8093f9ec54b0a0790c
I20260928 02:58:30.965330 15326 ts_tablet_manager.cc:1939] T dc7bedf8d5de4c8093f9ec54b0a0790c P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting consensus metadata
I20260928 02:58:30.965471 15325 log.cc:1196] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/dc7bedf8d5de4c8093f9ec54b0a0790c
I20260928 02:58:30.966096 15325 ts_tablet_manager.cc:1939] T dc7bedf8d5de4c8093f9ec54b0a0790c P 1631b004512b4e28847b971d97ce563a: Deleting consensus metadata
I20260928 02:58:30.967075 14758 catalog_manager.cc:4841] TS 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351): tablet dc7bedf8d5de4c8093f9ec54b0a0790c (table test-table [id=4cef1bc1d0664218b4e5bb80928d91a7]) successfully deleted
I20260928 02:58:30.970046 14758 catalog_manager.cc:4841] TS 1631b004512b4e28847b971d97ce563a (127.12.182.66:38673): tablet dc7bedf8d5de4c8093f9ec54b0a0790c (table test-table [id=4cef1bc1d0664218b4e5bb80928d91a7]) successfully deleted
I20260928 02:58:30.971863 15324 ts_tablet_manager.cc:1918] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:58:30.971998 15324 log.cc:1196] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/dc7bedf8d5de4c8093f9ec54b0a0790c
I20260928 02:58:30.972396 15324 ts_tablet_manager.cc:1939] T dc7bedf8d5de4c8093f9ec54b0a0790c P a950b7016e0d49ae874677d2ffb09265: Deleting consensus metadata
I20260928 02:58:30.973802 14757 catalog_manager.cc:4841] TS a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759): tablet dc7bedf8d5de4c8093f9ec54b0a0790c (table test-table [id=4cef1bc1d0664218b4e5bb80928d91a7]) successfully deleted
I20260928 02:58:30.989176 15283 raft_consensus.cc:482] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:30.989375 15283 raft_consensus.cc:504] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.990180 15283 leader_election.cc:288] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759), 1631b004512b4e28847b971d97ce563a (127.12.182.66:38673)
I20260928 02:58:30.990725 14927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3978c7e80842e9a4acaf83e614e79f" candidate_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a950b7016e0d49ae874677d2ffb09265" is_pre_election: true
I20260928 02:58:30.990875 15067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3978c7e80842e9a4acaf83e614e79f" candidate_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1631b004512b4e28847b971d97ce563a" is_pre_election: true
I20260928 02:58:30.990988 14927 raft_consensus.cc:2454] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d4e1bf13e24b6294d56c5ac33972c9 in term 0.
I20260928 02:58:30.991111 15067 raft_consensus.cc:2454] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d4e1bf13e24b6294d56c5ac33972c9 in term 0.
I20260928 02:58:30.991564 15140 leader_election.cc:302] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1631b004512b4e28847b971d97ce563a, 94d4e1bf13e24b6294d56c5ac33972c9; no voters:
I20260928 02:58:30.991878 15283 raft_consensus.cc:2781] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:30.992004 15283 raft_consensus.cc:482] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:30.992096 15283 raft_consensus.cc:3037] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:30.995186 15283 raft_consensus.cc:504] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:30.996444 15067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3978c7e80842e9a4acaf83e614e79f" candidate_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1631b004512b4e28847b971d97ce563a"
I20260928 02:58:30.996660 15067 raft_consensus.cc:3037] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:30.996783 14927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3978c7e80842e9a4acaf83e614e79f" candidate_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a950b7016e0d49ae874677d2ffb09265"
I20260928 02:58:30.996964 14927 raft_consensus.cc:3037] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:30.999891 14927 raft_consensus.cc:2454] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d4e1bf13e24b6294d56c5ac33972c9 in term 1.
I20260928 02:58:30.999944 15067 raft_consensus.cc:2454] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d4e1bf13e24b6294d56c5ac33972c9 in term 1.
I20260928 02:58:31.000393 15140 leader_election.cc:302] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1631b004512b4e28847b971d97ce563a, 94d4e1bf13e24b6294d56c5ac33972c9; no voters:
I20260928 02:58:31.000687 15318 raft_consensus.cc:2781] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:31.000830 15318 raft_consensus.cc:686] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 LEADER]: Becoming Leader. State: Replica: 94d4e1bf13e24b6294d56c5ac33972c9, State: Running, Role: LEADER
I20260928 02:58:31.001147 15318 consensus_queue.cc:237] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:31.001858 15283 leader_election.cc:288] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [CANDIDATE]: Term 1 election: Requested vote from peers a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759), 1631b004512b4e28847b971d97ce563a (127.12.182.66:38673)
I20260928 02:58:31.004118 14771 catalog_manager.cc:5495] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67). New cstate: current_term: 1 leader_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } health_report { overall_health: HEALTHY } } }
I20260928 02:58:31.043988 14927 raft_consensus.cc:1261] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Refusing update from remote peer 94d4e1bf13e24b6294d56c5ac33972c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:31.044791 15067 raft_consensus.cc:1261] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Refusing update from remote peer 94d4e1bf13e24b6294d56c5ac33972c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:31.045179 15283 consensus_queue.cc:1035] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:31.045430 15285 consensus_queue.cc:1035] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:58:31.085364 15304 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:58:32.680377 14771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:32.680583 14771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:32.684311 14771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a: Sending DeleteTablet for 3 replicas of tablet ea3978c7e80842e9a4acaf83e614e79f
I20260928 02:58:32.685074 14908 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3978c7e80842e9a4acaf83e614e79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:32 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:60046
I20260928 02:58:32.685150 15047 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3978c7e80842e9a4acaf83e614e79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:32 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:50832
I20260928 02:58:32.685168 15186 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3978c7e80842e9a4acaf83e614e79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:32 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:43920
I20260928 02:58:32.685762 15357 tablet_replica.cc:330] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265: stopping tablet replica
I20260928 02:58:32.685899 15359 tablet_replica.cc:330] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9: stopping tablet replica
I20260928 02:58:32.685951 15357 raft_consensus.cc:2229] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:32.686080 15359 raft_consensus.cc:2229] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:58:32.686216 15357 raft_consensus.cc:2258] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:32.686398 15359 raft_consensus.cc:2258] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:32.687111 15357 ts_tablet_manager.cc:1905] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:32.687569 15359 ts_tablet_manager.cc:1905] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:32.693506 15359 ts_tablet_manager.cc:1918] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:58:32.693539 15357 ts_tablet_manager.cc:1918] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:58:32.693645 15359 log.cc:1196] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/ea3978c7e80842e9a4acaf83e614e79f
I20260928 02:58:32.693661 15357 log.cc:1196] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/ea3978c7e80842e9a4acaf83e614e79f
I20260928 02:58:32.694052 15359 ts_tablet_manager.cc:1939] T ea3978c7e80842e9a4acaf83e614e79f P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting consensus metadata
I20260928 02:58:32.694154 15357 ts_tablet_manager.cc:1939] T ea3978c7e80842e9a4acaf83e614e79f P a950b7016e0d49ae874677d2ffb09265: Deleting consensus metadata
I20260928 02:58:32.695015 14758 catalog_manager.cc:4841] TS 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351): tablet ea3978c7e80842e9a4acaf83e614e79f (table test-table [id=f0f05f6ddee14f6bb22f554c6b74fdf3]) successfully deleted
I20260928 02:58:32.695264 15358 tablet_replica.cc:330] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a: stopping tablet replica
I20260928 02:58:32.695422 14757 catalog_manager.cc:4841] TS a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759): tablet ea3978c7e80842e9a4acaf83e614e79f (table test-table [id=f0f05f6ddee14f6bb22f554c6b74fdf3]) successfully deleted
I20260928 02:58:32.695448 15358 raft_consensus.cc:2229] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:32.695657 15358 raft_consensus.cc:2258] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:32.696375 15358 ts_tablet_manager.cc:1905] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:32.700824 15358 ts_tablet_manager.cc:1918] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:58:32.700932 15358 log.cc:1196] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/ea3978c7e80842e9a4acaf83e614e79f
I20260928 02:58:32.701274 15358 ts_tablet_manager.cc:1939] T ea3978c7e80842e9a4acaf83e614e79f P 1631b004512b4e28847b971d97ce563a: Deleting consensus metadata
I20260928 02:58:32.702116 14758 catalog_manager.cc:4841] TS 1631b004512b4e28847b971d97ce563a (127.12.182.66:38673): tablet ea3978c7e80842e9a4acaf83e614e79f (table test-table [id=f0f05f6ddee14f6bb22f554c6b74fdf3]) successfully deleted
Sep 28 02:58:32 dist-test-slave-k5rg krb5kdc[14726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:58:32 dist-test-slave-k5rg krb5kdc[14726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:32.730947 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:32.718292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.12.182.126:34775)
0928 02:58:32.718768 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:32.718773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:32.719082 (+ 309us) server_negotiation.cc:407] Connection header received
0928 02:58:32.719219 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:32.719223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:32.719289 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:32.719390 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:32.720556 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:32.721275 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:32.722473 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:32.722980 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:32.723070 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:32.726573 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:32.726578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:32.726581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:32.726613 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:32.728921 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:32.729493 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:32.729497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:32.729499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:32.729642 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:32.729913 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:32.729916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:32.729918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:32.730175 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:32.730345 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:32.730540 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 02:58:32.730692 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":221,"threads_started":1}
I20260928 02:58:32.734016 14771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:58:32.734606 14771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:32.740250 14908 tablet_service.cc:1467] Processing CreateTablet for tablet fd4c12f92a6440bdba354c7d109f0947 (DEFAULT_TABLE table=test-table [id=f6d41a18e1b14c739ed8337ab56817a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:32.740438 15047 tablet_service.cc:1467] Processing CreateTablet for tablet fd4c12f92a6440bdba354c7d109f0947 (DEFAULT_TABLE table=test-table [id=f6d41a18e1b14c739ed8337ab56817a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:32.740662 14908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4c12f92a6440bdba354c7d109f0947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:32.740658 15186 tablet_service.cc:1467] Processing CreateTablet for tablet fd4c12f92a6440bdba354c7d109f0947 (DEFAULT_TABLE table=test-table [id=f6d41a18e1b14c739ed8337ab56817a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:32.740805 15047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4c12f92a6440bdba354c7d109f0947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:32.741041 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4c12f92a6440bdba354c7d109f0947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:32.747170 15369 tablet_bootstrap.cc:492] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265: Bootstrap starting.
I20260928 02:58:32.747802 15368 tablet_bootstrap.cc:492] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9: Bootstrap starting.
I20260928 02:58:32.748554 15370 tablet_bootstrap.cc:492] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a: Bootstrap starting.
I20260928 02:58:32.749114 15369 tablet_bootstrap.cc:654] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:32.749330 15368 tablet_bootstrap.cc:654] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:32.750087 15370 tablet_bootstrap.cc:654] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:32.751307 15369 tablet_bootstrap.cc:492] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265: No bootstrap required, opened a new log
I20260928 02:58:32.751456 15368 tablet_bootstrap.cc:492] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9: No bootstrap required, opened a new log
I20260928 02:58:32.751451 15369 ts_tablet_manager.cc:1397] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260928 02:58:32.751579 15368 ts_tablet_manager.cc:1397] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:58:32.751914 15370 tablet_bootstrap.cc:492] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a: No bootstrap required, opened a new log
I20260928 02:58:32.752059 15370 ts_tablet_manager.cc:1397] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:58:32.752197 15369 raft_consensus.cc:348] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.752430 15369 raft_consensus.cc:374] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:32.752307 15368 raft_consensus.cc:348] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.752517 15368 raft_consensus.cc:374] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:32.752525 15369 raft_consensus.cc:729] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a950b7016e0d49ae874677d2ffb09265, State: Initialized, Role: FOLLOWER
I20260928 02:58:32.752604 15368 raft_consensus.cc:729] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4e1bf13e24b6294d56c5ac33972c9, State: Initialized, Role: FOLLOWER
I20260928 02:58:32.752696 15370 raft_consensus.cc:348] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.752740 15369 consensus_queue.cc:260] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.752892 15370 raft_consensus.cc:374] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:32.752816 15368 consensus_queue.cc:260] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.752976 15370 raft_consensus.cc:729] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1631b004512b4e28847b971d97ce563a, State: Initialized, Role: FOLLOWER
I20260928 02:58:32.753315 15368 ts_tablet_manager.cc:1428] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:58:32.753232 15370 consensus_queue.cc:260] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.753765 15370 ts_tablet_manager.cc:1428] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:58:32.754303 15369 ts_tablet_manager.cc:1428] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:58:32.757462 15352 raft_consensus.cc:482] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:32.757628 15352 raft_consensus.cc:504] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.758201 15352 leader_election.cc:288] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759), 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351)
I20260928 02:58:32.758566 14927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4c12f92a6440bdba354c7d109f0947" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a950b7016e0d49ae874677d2ffb09265" is_pre_election: true
I20260928 02:58:32.758597 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4c12f92a6440bdba354c7d109f0947" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" is_pre_election: true
I20260928 02:58:32.758749 14927 raft_consensus.cc:2454] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 0.
I20260928 02:58:32.758816 15205 raft_consensus.cc:2454] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 0.
I20260928 02:58:32.759124 15000 leader_election.cc:302] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1631b004512b4e28847b971d97ce563a, a950b7016e0d49ae874677d2ffb09265; no voters:
I20260928 02:58:32.759352 15352 raft_consensus.cc:2781] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:32.759445 15352 raft_consensus.cc:482] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:32.759524 15352 raft_consensus.cc:3037] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:32.761296 15352 raft_consensus.cc:504] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.761785 15352 leader_election.cc:288] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 election: Requested vote from peers a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759), 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351)
I20260928 02:58:32.762180 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4c12f92a6440bdba354c7d109f0947" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4e1bf13e24b6294d56c5ac33972c9"
I20260928 02:58:32.762221 14927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4c12f92a6440bdba354c7d109f0947" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a950b7016e0d49ae874677d2ffb09265"
I20260928 02:58:32.762333 15205 raft_consensus.cc:3037] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:32.762408 14927 raft_consensus.cc:3037] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:32.764197 15205 raft_consensus.cc:2454] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 1.
I20260928 02:58:32.764245 14927 raft_consensus.cc:2454] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 1.
I20260928 02:58:32.764626 15000 leader_election.cc:302] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1631b004512b4e28847b971d97ce563a, 94d4e1bf13e24b6294d56c5ac33972c9, a950b7016e0d49ae874677d2ffb09265; no voters:
I20260928 02:58:32.764885 15352 raft_consensus.cc:2781] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:32.765027 15352 raft_consensus.cc:686] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 1 LEADER]: Becoming Leader. State: Replica: 1631b004512b4e28847b971d97ce563a, State: Running, Role: LEADER
I20260928 02:58:32.765278 15352 consensus_queue.cc:237] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.767418 14770 catalog_manager.cc:5495] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1631b004512b4e28847b971d97ce563a (127.12.182.66). New cstate: current_term: 1 leader_uuid: "1631b004512b4e28847b971d97ce563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } health_report { overall_health: UNKNOWN } } }
I20260928 02:58:32.787218 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:32.782267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.12.182.66:38673)
0928 02:58:32.782716 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:32.782720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:32.782738 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:58:32.782855 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:32.782860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:32.782917 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:32.782992 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:32.783366 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:32.784096 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:32.785500 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:32.785968 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:32.786073 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:32.786495 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:32.786613 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:32.786683 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:32.786842 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:58:32.786915 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":260,"threads_started":1}
I20260928 02:58:32.801318 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:32.796894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:35783 (local address 127.12.182.66:38673)
0928 02:58:32.797072 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:32.797079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:32.797361 (+ 282us) server_negotiation.cc:407] Connection header received
0928 02:58:32.797480 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:32.797483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:32.797540 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:32.797627 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:32.798005 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:32.798815 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:32.800037 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:32.800428 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:32.800515 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:32.800820 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:32.800908 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:32.800968 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:32.801048 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:58:32.801103 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:58:32.801380 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:32.796828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:38673 (local address 127.12.182.66:35783)
0928 02:58:32.797277 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:32.797291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:32.797389 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:32.797652 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:32.797657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:32.797669 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:58:32.797867 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:32.797874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:32.798921 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:32.798925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:32.799908 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:32.799919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:32.800557 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:32.800561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:32.800682 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:32.800713 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:58:32.800971 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:58:32.800978 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:58:32.801027 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:58:32.801117 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":255,"threads_started":1}
I20260928 02:58:32.808351 15205 raft_consensus.cc:1261] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Refusing update from remote peer 1631b004512b4e28847b971d97ce563a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:32.808447 14927 raft_consensus.cc:1261] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Refusing update from remote peer 1631b004512b4e28847b971d97ce563a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:32.808919 15334 consensus_queue.cc:1035] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:32.809177 15352 consensus_queue.cc:1035] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:32.850483 14769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:32.850696 14769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:32.853545 14769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a: Sending DeleteTablet for 3 replicas of tablet fd4c12f92a6440bdba354c7d109f0947
I20260928 02:58:32.854295 14908 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4c12f92a6440bdba354c7d109f0947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:32 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:60046
I20260928 02:58:32.854470 15046 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4c12f92a6440bdba354c7d109f0947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:32 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:50832
I20260928 02:58:32.854560 15357 tablet_replica.cc:330] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265: stopping tablet replica
I20260928 02:58:32.854717 15358 tablet_replica.cc:330] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a: stopping tablet replica
I20260928 02:58:32.854748 15357 raft_consensus.cc:2229] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:32.854873 15358 raft_consensus.cc:2229] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:58:32.854923 15357 raft_consensus.cc:2258] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:32.855211 15186 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4c12f92a6440bdba354c7d109f0947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:32 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:43920
I20260928 02:58:32.855265 15358 raft_consensus.cc:2258] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:32.855468 15359 tablet_replica.cc:330] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9: stopping tablet replica
I20260928 02:58:32.855648 15359 raft_consensus.cc:2229] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:32.855813 15359 raft_consensus.cc:2258] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:32.855943 15357 ts_tablet_manager.cc:1905] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:32.857225 15359 ts_tablet_manager.cc:1905] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:32.860435 15357 ts_tablet_manager.cc:1918] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:58:32.860536 15357 log.cc:1196] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/fd4c12f92a6440bdba354c7d109f0947
I20260928 02:58:32.860798 15357 ts_tablet_manager.cc:1939] T fd4c12f92a6440bdba354c7d109f0947 P a950b7016e0d49ae874677d2ffb09265: Deleting consensus metadata
I20260928 02:58:32.861706 14757 catalog_manager.cc:4841] TS a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759): tablet fd4c12f92a6440bdba354c7d109f0947 (table test-table [id=f6d41a18e1b14c739ed8337ab56817a2]) successfully deleted
I20260928 02:58:32.862326 15358 ts_tablet_manager.cc:1905] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:32.862167 14769 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:58:32.862884 14769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:32.863526 15359 ts_tablet_manager.cc:1918] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:58:32.863662 15359 log.cc:1196] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/fd4c12f92a6440bdba354c7d109f0947
I20260928 02:58:32.864066 15359 ts_tablet_manager.cc:1939] T fd4c12f92a6440bdba354c7d109f0947 P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting consensus metadata
I20260928 02:58:32.865082 14758 catalog_manager.cc:4841] TS 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351): tablet fd4c12f92a6440bdba354c7d109f0947 (table test-table [id=f6d41a18e1b14c739ed8337ab56817a2]) successfully deleted
I20260928 02:58:32.869655 15358 ts_tablet_manager.cc:1918] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:58:32.869817 15358 log.cc:1196] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/fd4c12f92a6440bdba354c7d109f0947
I20260928 02:58:32.870162 14908 tablet_service.cc:1467] Processing CreateTablet for tablet 9308d2d0ec7340c1a155db69428d19cb (DEFAULT_TABLE table=test-table [id=ba40b3580c054c22b193380eca66b957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:32.870584 14908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9308d2d0ec7340c1a155db69428d19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:32.870524 15186 tablet_service.cc:1467] Processing CreateTablet for tablet 9308d2d0ec7340c1a155db69428d19cb (DEFAULT_TABLE table=test-table [id=ba40b3580c054c22b193380eca66b957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:32.870903 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9308d2d0ec7340c1a155db69428d19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:32.873929 15046 tablet_service.cc:1467] Processing CreateTablet for tablet 9308d2d0ec7340c1a155db69428d19cb (DEFAULT_TABLE table=test-table [id=ba40b3580c054c22b193380eca66b957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:32.874240 15046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9308d2d0ec7340c1a155db69428d19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:32.876827 15358 ts_tablet_manager.cc:1939] T fd4c12f92a6440bdba354c7d109f0947 P 1631b004512b4e28847b971d97ce563a: Deleting consensus metadata
I20260928 02:58:32.876914 15368 tablet_bootstrap.cc:492] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9: Bootstrap starting.
I20260928 02:58:32.877985 14758 catalog_manager.cc:4841] TS 1631b004512b4e28847b971d97ce563a (127.12.182.66:38673): tablet fd4c12f92a6440bdba354c7d109f0947 (table test-table [id=f6d41a18e1b14c739ed8337ab56817a2]) successfully deleted
I20260928 02:58:32.878526 15368 tablet_bootstrap.cc:654] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:32.880625 15368 tablet_bootstrap.cc:492] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9: No bootstrap required, opened a new log
I20260928 02:58:32.880790 15368 ts_tablet_manager.cc:1397] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:58:32.880867 15370 tablet_bootstrap.cc:492] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a: Bootstrap starting.
I20260928 02:58:32.881538 15368 raft_consensus.cc:348] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.881839 15368 raft_consensus.cc:374] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:32.881919 15368 raft_consensus.cc:729] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4e1bf13e24b6294d56c5ac33972c9, State: Initialized, Role: FOLLOWER
I20260928 02:58:32.882248 15369 tablet_bootstrap.cc:492] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265: Bootstrap starting.
I20260928 02:58:32.882212 15368 consensus_queue.cc:260] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.882547 15370 tablet_bootstrap.cc:654] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:32.882800 15368 ts_tablet_manager.cc:1428] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:58:32.883819 15369 tablet_bootstrap.cc:654] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:32.886068 15369 tablet_bootstrap.cc:492] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265: No bootstrap required, opened a new log
I20260928 02:58:32.886219 15369 ts_tablet_manager.cc:1397] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 02:58:32.886930 15369 raft_consensus.cc:348] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.887180 15369 raft_consensus.cc:374] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:32.887241 15369 raft_consensus.cc:729] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a950b7016e0d49ae874677d2ffb09265, State: Initialized, Role: FOLLOWER
I20260928 02:58:32.887471 15369 consensus_queue.cc:260] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.887985 15369 ts_tablet_manager.cc:1428] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:58:32.889581 15370 tablet_bootstrap.cc:492] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a: No bootstrap required, opened a new log
I20260928 02:58:32.889787 15370 ts_tablet_manager.cc:1397] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260928 02:58:32.890561 15370 raft_consensus.cc:348] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.890801 15370 raft_consensus.cc:374] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:32.890887 15370 raft_consensus.cc:729] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1631b004512b4e28847b971d97ce563a, State: Initialized, Role: FOLLOWER
I20260928 02:58:32.891126 15370 consensus_queue.cc:260] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.891667 15370 ts_tablet_manager.cc:1428] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:58:32.989719 15334 raft_consensus.cc:482] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:32.989977 15334 raft_consensus.cc:504] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.990713 15334 leader_election.cc:288] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759), 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351)
I20260928 02:58:32.991292 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9308d2d0ec7340c1a155db69428d19cb" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" is_pre_election: true
I20260928 02:58:32.991313 14928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9308d2d0ec7340c1a155db69428d19cb" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a950b7016e0d49ae874677d2ffb09265" is_pre_election: true
I20260928 02:58:32.991536 15205 raft_consensus.cc:2454] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 0.
I20260928 02:58:32.991551 14928 raft_consensus.cc:2454] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 0.
I20260928 02:58:32.991943 15000 leader_election.cc:302] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1631b004512b4e28847b971d97ce563a, a950b7016e0d49ae874677d2ffb09265; no voters:
I20260928 02:58:32.992234 15334 raft_consensus.cc:2781] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:32.992333 15334 raft_consensus.cc:482] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:32.992444 15334 raft_consensus.cc:3037] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:32.994685 15334 raft_consensus.cc:504] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:32.995229 15334 leader_election.cc:288] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 election: Requested vote from peers a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759), 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351)
I20260928 02:58:32.995718 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9308d2d0ec7340c1a155db69428d19cb" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4e1bf13e24b6294d56c5ac33972c9"
I20260928 02:58:32.995718 14928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9308d2d0ec7340c1a155db69428d19cb" candidate_uuid: "1631b004512b4e28847b971d97ce563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a950b7016e0d49ae874677d2ffb09265"
I20260928 02:58:32.995916 15205 raft_consensus.cc:3037] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:32.995916 14928 raft_consensus.cc:3037] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:32.998775 15205 raft_consensus.cc:2454] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 1.
I20260928 02:58:32.998775 14928 raft_consensus.cc:2454] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1631b004512b4e28847b971d97ce563a in term 1.
I20260928 02:58:32.999220 15001 leader_election.cc:302] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1631b004512b4e28847b971d97ce563a, 94d4e1bf13e24b6294d56c5ac33972c9; no voters:
I20260928 02:58:32.999450 15334 raft_consensus.cc:2781] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:32.999655 15334 raft_consensus.cc:686] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 1 LEADER]: Becoming Leader. State: Replica: 1631b004512b4e28847b971d97ce563a, State: Running, Role: LEADER
I20260928 02:58:32.999948 15334 consensus_queue.cc:237] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } }
I20260928 02:58:33.002856 14769 catalog_manager.cc:5495] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1631b004512b4e28847b971d97ce563a (127.12.182.66). New cstate: current_term: 1 leader_uuid: "1631b004512b4e28847b971d97ce563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1631b004512b4e28847b971d97ce563a" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 38673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 } health_report { overall_health: UNKNOWN } } }
I20260928 02:58:33.051637 14928 raft_consensus.cc:1261] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Refusing update from remote peer 1631b004512b4e28847b971d97ce563a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:33.051955 15205 raft_consensus.cc:1261] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Refusing update from remote peer 1631b004512b4e28847b971d97ce563a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:33.052198 15337 consensus_queue.cc:1035] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a950b7016e0d49ae874677d2ffb09265" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:33.052541 15352 consensus_queue.cc:1035] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d4e1bf13e24b6294d56c5ac33972c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:33.058115 15339 mvcc.cc:205] Tried to move back new op lower bound from 7334151426254057472 to 7334151426050252800. Current Snapshot: MvccSnapshot[applied={T|T < 7334151426254057472}]
W20260928 02:58:33.072176 15335 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:58:34.671558 14769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:34.671778 14769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:34.675638 14769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca9ee76bce504f80b214e312c22c421a: Sending DeleteTablet for 3 replicas of tablet 9308d2d0ec7340c1a155db69428d19cb
I20260928 02:58:34.676364 14908 tablet_service.cc:1514] Processing DeleteTablet for tablet 9308d2d0ec7340c1a155db69428d19cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:34 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:60046
I20260928 02:58:34.676383 15046 tablet_service.cc:1514] Processing DeleteTablet for tablet 9308d2d0ec7340c1a155db69428d19cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:34 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:50832
I20260928 02:58:34.676569 15186 tablet_service.cc:1514] Processing DeleteTablet for tablet 9308d2d0ec7340c1a155db69428d19cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:34 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:43920
I20260928 02:58:34.677096 15413 tablet_replica.cc:330] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265: stopping tablet replica
I20260928 02:58:34.677251 15412 tablet_replica.cc:330] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a: stopping tablet replica
I20260928 02:58:34.677281 15413 raft_consensus.cc:2229] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:34.677424 15412 raft_consensus.cc:2229] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:58:34.677485 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14842
I20260928 02:58:34.677520 15413 raft_consensus.cc:2258] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:34.677798 15412 raft_consensus.cc:2258] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:34.678486 15413 ts_tablet_manager.cc:1905] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:34.686285 15413 ts_tablet_manager.cc:1918] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:58:34.686493 15413 log.cc:1196] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/9308d2d0ec7340c1a155db69428d19cb
I20260928 02:58:34.686667 15412 ts_tablet_manager.cc:1905] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:34.687045 15413 ts_tablet_manager.cc:1939] T 9308d2d0ec7340c1a155db69428d19cb P a950b7016e0d49ae874677d2ffb09265: Deleting consensus metadata
I20260928 02:58:34.690315 14757 catalog_manager.cc:4841] TS a950b7016e0d49ae874677d2ffb09265 (127.12.182.65:39759): tablet 9308d2d0ec7340c1a155db69428d19cb (table test-table [id=ba40b3580c054c22b193380eca66b957]) successfully deleted
I20260928 02:58:34.692982 15412 ts_tablet_manager.cc:1918] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:58:34.693109 15412 log.cc:1196] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/9308d2d0ec7340c1a155db69428d19cb
I20260928 02:58:34.693444 15412 ts_tablet_manager.cc:1939] T 9308d2d0ec7340c1a155db69428d19cb P 1631b004512b4e28847b971d97ce563a: Deleting consensus metadata
I20260928 02:58:34.694041 15414 tablet_replica.cc:330] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9: stopping tablet replica
I20260928 02:58:34.694279 15414 raft_consensus.cc:2229] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:34.694494 14758 catalog_manager.cc:4841] TS 1631b004512b4e28847b971d97ce563a (127.12.182.66:38673): tablet 9308d2d0ec7340c1a155db69428d19cb (table test-table [id=ba40b3580c054c22b193380eca66b957]) successfully deleted
I20260928 02:58:34.694585 15414 raft_consensus.cc:2258] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:34.695497 15414 ts_tablet_manager.cc:1905] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:58:34 dist-test-slave-k5rg krb5kdc[14726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:58:34.700217 15414 ts_tablet_manager.cc:1918] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:58:34.700327 15414 log.cc:1196] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/9308d2d0ec7340c1a155db69428d19cb
I20260928 02:58:34.700664 15414 ts_tablet_manager.cc:1939] T 9308d2d0ec7340c1a155db69428d19cb P 94d4e1bf13e24b6294d56c5ac33972c9: Deleting consensus metadata
I20260928 02:58:34.701508 14758 catalog_manager.cc:4841] TS 94d4e1bf13e24b6294d56c5ac33972c9 (127.12.182.67:33351): tablet 9308d2d0ec7340c1a155db69428d19cb (table test-table [id=ba40b3580c054c22b193380eca66b957]) successfully deleted
I20260928 02:58:34.704658 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:34.678343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.12.182.65:39759)
0928 02:58:34.690292 (+ 11949us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:34.690296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:34.690309 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:58:34.690351 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:34.690355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:34.690433 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:34.690527 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:34.691768 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:34.692872 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:34.694454 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:34.694947 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:34.695050 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:34.699372 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:34.699386 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:34.699401 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:34.699434 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:34.702314 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:34.702921 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:34.702928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:34.702930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:34.703066 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:34.703405 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:34.703409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:34.703411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:34.703755 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:34.704014 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:34.704168 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:58:34.704330 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11833,"thread_start_us":11710,"threads_started":1}
W20260928 02:58:34.705264 14970 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45656
W20260928 02:58:34.706001 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14842: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:58:34.706197 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 14842
I20260928 02:58:34.720093 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14982
Sep 28 02:58:34 dist-test-slave-k5rg krb5kdc[14726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 02:58:34.733467 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:34.720742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.12.182.66:38673)
0928 02:58:34.721195 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:34.721200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:34.721215 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:58:34.721266 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:34.721270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:34.721331 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:34.721431 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:34.722630 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:34.723380 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:34.724621 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:34.725052 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:34.725148 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:34.728817 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:34.728829 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:34.728845 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:34.728877 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:34.731339 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:34.731915 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:34.731921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:34.731924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:34.732043 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:34.732313 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:34.732316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:34.732318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:34.732655 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:34.732866 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:34.733053 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:58:34.733214 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":224,"threads_started":1}
W20260928 02:58:34.734094 15112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56350
W20260928 02:58:34.734709 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14982: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:58:34.734841 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 14982
I20260928 02:58:34.754290 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 15121
Sep 28 02:58:34 dist-test-slave-k5rg krb5kdc[14726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 02:58:34.768127 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:34.754966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.12.182.67:33351)
0928 02:58:34.755465 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:34.755470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:34.755486 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:58:34.755537 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:34.755540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:34.755609 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:34.755697 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:34.756849 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:34.757578 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:34.758815 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:34.759205 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:34.759297 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:34.763131 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:34.763145 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:34.763161 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:34.763191 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:34.765908 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:34.766506 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:34.766513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:34.766515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:34.766641 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:34.766927 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:34.766930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:34.766932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:34.767275 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:34.767501 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:34.767708 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:58:34.767877 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
W20260928 02:58:34.768757 15251 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59892
W20260928 02:58:34.769307 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 15121: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:58:34.769467 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 15121
I20260928 02:58:34.786643 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 14739
I20260928 02:58:34.787933 14811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:34.902657 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 14739
2026-09-28T02:58:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7917 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:34 dist-test-slave-k5rg krb5kdc[15422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:34 dist-test-slave-k5rg krb5kdc[15422](info): set up 2 sockets
Sep 28 02:58:34 dist-test-slave-k5rg krb5kdc[15422](info): commencing operation
krb5kdc: starting...
W20260928 02:58:35.889739 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:58:36 dist-test-slave-k5rg krb5kdc[15422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:58:36.053166 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:38475
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:37661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:38475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:36.356068 15435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:36.356359 15435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:36.356487 15435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:36.363691 15435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:36.363809 15435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:36.363902 15435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:36.363977 15435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:36.364043 15435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:36.373574 15435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:37661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:38475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:38475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:36.374789 15435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:36.375994 15435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:36.386852 15440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:36.386880 15443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:36.386972 15441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:36.387296 15435 server_base.cc:1034] running on GCE node
I20260928 02:58:36.388299 15435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:36.389588 15435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:36.390826 15435 hybrid_clock.cc:648] HybridClock initialized: now 1790564316390776 us; error 42 us; skew 500 ppm
I20260928 02:58:36.391139 15435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:36 dist-test-slave-k5rg krb5kdc[15422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564316, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:36.396952 15435 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:58:36.398473 15435 webserver.cc:460] Webserver started at http://127.12.182.126:39589/ using document root <none> and password file <none>
I20260928 02:58:36.399147 15435 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:36.399260 15435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:36.399540 15435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:36.402456 15435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "2b32a37e34b3443ea02d7cac528a348c"
format_stamp: "Formatted at 2026-09-28 02:58:36 on dist-test-slave-k5rg"
server_key: "1b76908c7295b3f1a515d9a50dc3b07e"
server_key_iv: "db1e67b431181d4eb878e10290f9f155"
server_key_version: "encryptionkey@0"
I20260928 02:58:36.403168 15435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "2b32a37e34b3443ea02d7cac528a348c"
format_stamp: "Formatted at 2026-09-28 02:58:36 on dist-test-slave-k5rg"
server_key: "1b76908c7295b3f1a515d9a50dc3b07e"
server_key_iv: "db1e67b431181d4eb878e10290f9f155"
server_key_version: "encryptionkey@0"
I20260928 02:58:36.408100 15435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:58:36.411326 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:36.412540 15435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 02:58:36.412693 15435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "2b32a37e34b3443ea02d7cac528a348c"
format_stamp: "Formatted at 2026-09-28 02:58:36 on dist-test-slave-k5rg"
server_key: "1b76908c7295b3f1a515d9a50dc3b07e"
server_key_iv: "db1e67b431181d4eb878e10290f9f155"
server_key_version: "encryptionkey@0"
I20260928 02:58:36.412859 15435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:36.427266 15435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:36.430933 15435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:36.431181 15435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:36.447832 15435 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:38475
I20260928 02:58:36.447839 15501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:38475 every 8 connection(s)
I20260928 02:58:36.449121 15435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:36.452620 15502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:36.457655 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 15435
I20260928 02:58:36.457855 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:36.458462 13017 external_mini_cluster.cc:1428] Setting key 315cbaa658bf99db8f3ff38f27e99a54
I20260928 02:58:36.462558 15502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c: Bootstrap starting.
I20260928 02:58:36.465936 15502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:36.466941 15502 log.cc:824] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:36.469712 15502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c: No bootstrap required, opened a new log
Sep 28 02:58:36 dist-test-slave-k5rg krb5kdc[15422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:36.477308 15502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b32a37e34b3443ea02d7cac528a348c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38475 } }
I20260928 02:58:36.477671 15502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:36.477766 15502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b32a37e34b3443ea02d7cac528a348c, State: Initialized, Role: FOLLOWER
I20260928 02:58:36.478376 15502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b32a37e34b3443ea02d7cac528a348c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38475 } }
I20260928 02:58:36.478608 15502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:36.478765 15502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:36.478927 15502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:36.479890 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:36.460347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.12.182.126:38475)
0928 02:58:36.461095 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:36.461106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:36.461143 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:58:36.462802 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:36.462824 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:36.463194 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:36.463486 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:36.465192 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:36.466520 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:36.468018 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:36.468653 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:36.468776 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:36.472999 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:36.473006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:36.473020 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:36.473066 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:36.475950 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:36.476639 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:36.476645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:36.476651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:36.476810 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:36.477155 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:36.477158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:36.477160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:36.477751 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:36.478027 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:36.478439 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 02:58:36.478858 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":332,"threads_started":1}
I20260928 02:58:36.481262 15502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b32a37e34b3443ea02d7cac528a348c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38475 } }
I20260928 02:58:36.481868 15502 leader_election.cc:302] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b32a37e34b3443ea02d7cac528a348c; no voters:
I20260928 02:58:36.482455 15502 leader_election.cc:288] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:36.482587 15507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:36.483233 15507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [term 1 LEADER]: Becoming Leader. State: Replica: 2b32a37e34b3443ea02d7cac528a348c, State: Running, Role: LEADER
I20260928 02:58:36.483677 15507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b32a37e34b3443ea02d7cac528a348c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38475 } }
I20260928 02:58:36.484392 15502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:36.486552 15509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b32a37e34b3443ea02d7cac528a348c. Latest consensus state: current_term: 1 leader_uuid: "2b32a37e34b3443ea02d7cac528a348c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b32a37e34b3443ea02d7cac528a348c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38475 } } }
I20260928 02:58:36.486963 15509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:36.487071 15508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b32a37e34b3443ea02d7cac528a348c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b32a37e34b3443ea02d7cac528a348c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38475 } } }
I20260928 02:58:36.487353 15508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:36.488988 15513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:36.501200 15513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:36.509848 15513 catalog_manager.cc:1347] Generated new cluster ID: 1377277cddbf4241a333b9ff4d59d83c
I20260928 02:58:36.509951 15513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:36.572183 15513 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:36.573472 15513 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:36.582881 15513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b32a37e34b3443ea02d7cac528a348c: Generated new TSK 0
I20260928 02:58:36.583679 15513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:58:36.629426 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:38475
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:37661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:36.935328 15528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:36.935591 15528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:36.935707 15528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:36.942898 15528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:36.943001 15528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:36.943140 15528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:58:36.952488 15528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:37661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:38475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:36.953660 15528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:36.954902 15528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:36.967125 15533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:36.967088 15536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:36.967149 15534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:36.967919 15528 server_base.cc:1034] running on GCE node
I20260928 02:58:36.968495 15528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:36.969362 15528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:36.970574 15528 hybrid_clock.cc:648] HybridClock initialized: now 1790564316970519 us; error 46 us; skew 500 ppm
I20260928 02:58:36.970875 15528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:36 dist-test-slave-k5rg krb5kdc[15422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564316, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:36.973739 15528 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:58:36.975054 15528 webserver.cc:460] Webserver started at http://127.12.182.65:44495/ using document root <none> and password file <none>
I20260928 02:58:36.975685 15528 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:36.975795 15528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:36.976068 15528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:36.978852 15528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "3d68c6a32e224f83afca08e821769ff1"
format_stamp: "Formatted at 2026-09-28 02:58:36 on dist-test-slave-k5rg"
server_key: "f1508012574e80b5d517acdd7e26cf34"
server_key_iv: "98f20f5f5ad3abf3d2737697616c8292"
server_key_version: "encryptionkey@0"
I20260928 02:58:36.979583 15528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "3d68c6a32e224f83afca08e821769ff1"
format_stamp: "Formatted at 2026-09-28 02:58:36 on dist-test-slave-k5rg"
server_key: "f1508012574e80b5d517acdd7e26cf34"
server_key_iv: "98f20f5f5ad3abf3d2737697616c8292"
server_key_version: "encryptionkey@0"
I20260928 02:58:36.984349 15528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:58:36.987646 15543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:36.988855 15528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:58:36.989014 15528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "3d68c6a32e224f83afca08e821769ff1"
format_stamp: "Formatted at 2026-09-28 02:58:36 on dist-test-slave-k5rg"
server_key: "f1508012574e80b5d517acdd7e26cf34"
server_key_iv: "98f20f5f5ad3abf3d2737697616c8292"
server_key_version: "encryptionkey@0"
I20260928 02:58:36.989181 15528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:37.006938 15528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:37.010699 15528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:37.010959 15528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:37.011849 15528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:37.013098 15528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:37.013173 15528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:37.013294 15528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:37.013346 15528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:37.042256 15528 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:45053
I20260928 02:58:37.042300 15655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:45053 every 8 connection(s)
I20260928 02:58:37.043424 15528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:58:37.045995 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 15528
I20260928 02:58:37.046198 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:58:37.046983 13017 external_mini_cluster.cc:1428] Setting key db7aaa387d64aa9fff3d86f7540ce51e
Sep 28 02:58:37 dist-test-slave-k5rg krb5kdc[15422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564316, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:37.063371 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:37.045680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:40741 (local address 127.12.182.126:38475)
0928 02:58:37.046077 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:37.046081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:37.047227 (+ 1146us) server_negotiation.cc:407] Connection header received
0928 02:58:37.049396 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:37.049402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:37.049477 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:37.049566 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:37.050987 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.051691 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.053229 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.053692 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.053804 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:37.057219 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:37.057225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:37.057230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:37.057266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:37.060319 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:37.060984 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:37.060989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:37.060993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:37.061144 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:37.061516 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:37.061521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:37.061524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:37.061917 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:37.062069 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:37.062700 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 02:58:37.062930 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":194,"threads_started":1}
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
I20260928 02:58:37.064040 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:37.046346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:38475 (local address 127.12.182.65:40741)
0928 02:58:37.047072 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:37.047125 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:37.049158 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:37.049822 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:37.049830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:37.050302 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:37.050837 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:37.050854 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.051834 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.051838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:37.053098 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:37.053114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.053916 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.053923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:37.054194 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:37.054808 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:37.054834 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:37.057009 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:37.060461 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:37.060467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:37.060480 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:37.060871 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:37.061238 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:37.061241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:37.061243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:37.061415 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:37.062022 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:37.062028 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:37.062345 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 02:58:37.062572 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 02:58:37.062948 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":292,"threads_started":1}
I20260928 02:58:37.065428 15656 heartbeater.cc:347] Connected to a master server at 127.12.182.126:38475
I20260928 02:58:37.065775 15656 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
I20260928 02:58:37.066612 15656 heartbeater.cc:511] Master 127.12.182.126:38475 requested a full tablet report, sending...
I20260928 02:58:37.068912 15467 ts_manager.cc:194] Registered new tserver with Master: 3d68c6a32e224f83afca08e821769ff1 (127.12.182.65:45053)
I20260928 02:58:37.070824 15467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:40741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:58:37.087741 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:38475
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:37661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:37.389864 15662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:37.390148 15662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:37.390275 15662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:37.397328 15662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:37.397444 15662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:37.397593 15662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:58:37.407109 15662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:37661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:38475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:37.408325 15662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:37.409513 15662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:37.421571 15667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:37.421499 15670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:37.421535 15668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:37.422230 15662 server_base.cc:1034] running on GCE node
I20260928 02:58:37.422768 15662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:37.423604 15662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:37.424803 15662 hybrid_clock.cc:648] HybridClock initialized: now 1790564317424756 us; error 33 us; skew 500 ppm
I20260928 02:58:37.425089 15662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:37 dist-test-slave-k5rg krb5kdc[15422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:37.427858 15662 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:58:37.429211 15662 webserver.cc:460] Webserver started at http://127.12.182.66:45879/ using document root <none> and password file <none>
I20260928 02:58:37.429909 15662 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:37.430020 15662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:37.430285 15662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:37.432986 15662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "b8718d3ee51b4053af3f00a304856875"
format_stamp: "Formatted at 2026-09-28 02:58:37 on dist-test-slave-k5rg"
server_key: "da923b583a0aba54a063aabdbdeae8f2"
server_key_iv: "759157fa35dddbd94d1c59a00f98ed7c"
server_key_version: "encryptionkey@0"
I20260928 02:58:37.433758 15662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "b8718d3ee51b4053af3f00a304856875"
format_stamp: "Formatted at 2026-09-28 02:58:37 on dist-test-slave-k5rg"
server_key: "da923b583a0aba54a063aabdbdeae8f2"
server_key_iv: "759157fa35dddbd94d1c59a00f98ed7c"
server_key_version: "encryptionkey@0"
I20260928 02:58:37.438441 15662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:58:37.441591 15677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:37.442821 15662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:58:37.442962 15662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "b8718d3ee51b4053af3f00a304856875"
format_stamp: "Formatted at 2026-09-28 02:58:37 on dist-test-slave-k5rg"
server_key: "da923b583a0aba54a063aabdbdeae8f2"
server_key_iv: "759157fa35dddbd94d1c59a00f98ed7c"
server_key_version: "encryptionkey@0"
I20260928 02:58:37.443125 15662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:37.469775 15662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:37.473557 15662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:37.473839 15662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:37.474735 15662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:37.475979 15662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:37.476053 15662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:37.476166 15662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:37.476215 15662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:37.505210 15662 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:33245
I20260928 02:58:37.505252 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:33245 every 8 connection(s)
I20260928 02:58:37.506485 15662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:58:37.512377 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 15662
I20260928 02:58:37.512566 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:58:37.513092 13017 external_mini_cluster.cc:1428] Setting key f0b811721020907e8a49809797c0c2d8
Sep 28 02:58:37 dist-test-slave-k5rg krb5kdc[15422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:37.526714 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:37.508771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:41929 (local address 127.12.182.126:38475)
0928 02:58:37.508974 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:37.508981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:37.510065 (+ 1084us) server_negotiation.cc:407] Connection header received
0928 02:58:37.511812 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:37.511816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:37.511879 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:37.511963 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:37.513847 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.514588 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.516692 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.517191 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.517288 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:37.520896 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:37.520903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:37.520907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:37.520945 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:37.523948 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:37.524596 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:37.524602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:37.524605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:37.524751 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:37.525108 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:37.525112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:37.525115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:37.525401 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:37.525540 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:37.526172 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 02:58:37.526383 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:58:37.527488 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:37.509248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:38475 (local address 127.12.182.66:41929)
0928 02:58:37.509916 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:37.509958 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:37.511580 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:37.512196 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:37.512208 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:37.512903 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:37.513688 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:37.513710 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.514734 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.514738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:37.516528 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:37.516550 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.517401 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.517407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:37.517691 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:37.518340 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:37.518367 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:37.520688 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:37.524071 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:37.524077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:37.524089 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:37.524471 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:37.524839 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:37.524842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:37.524844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:37.525007 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:37.525513 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:37.525518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:37.525840 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 02:58:37.526069 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 02:58:37.526432 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":274,"threads_started":1}
I20260928 02:58:37.528888 15790 heartbeater.cc:347] Connected to a master server at 127.12.182.126:38475
I20260928 02:58:37.529223 15790 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:37.530100 15790 heartbeater.cc:511] Master 127.12.182.126:38475 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
I20260928 02:58:37.531629 15467 ts_manager.cc:194] Registered new tserver with Master: b8718d3ee51b4053af3f00a304856875 (127.12.182.66:33245)
I20260928 02:58:37.532430 15467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:41929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:58:37.552345 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:38475
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:37661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:37.849303 15795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:37.849550 15795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:37.849654 15795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:37.856418 15795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:37.856505 15795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:37.856629 15795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:58:37.865665 15795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:37661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:38475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:37.866753 15795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:37.867931 15795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:37.879758 15800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:37.879804 15803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:37.879825 15801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:37.880395 15795 server_base.cc:1034] running on GCE node
I20260928 02:58:37.880962 15795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:37.881814 15795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:37.882998 15795 hybrid_clock.cc:648] HybridClock initialized: now 1790564317882950 us; error 43 us; skew 500 ppm
I20260928 02:58:37.883275 15795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:37 dist-test-slave-k5rg krb5kdc[15422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:37.886142 15795 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:58:37.887434 15795 webserver.cc:460] Webserver started at http://127.12.182.67:43903/ using document root <none> and password file <none>
I20260928 02:58:37.888048 15795 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:37.888152 15795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:37.888427 15795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:37.891185 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "fffdab5d22c64585ac469a30c985b66c"
format_stamp: "Formatted at 2026-09-28 02:58:37 on dist-test-slave-k5rg"
server_key: "6c8c957342ee140facf1fe7c27cb2d1e"
server_key_iv: "4e3b3bef24894a4c2fde6a7ea140ab8b"
server_key_version: "encryptionkey@0"
I20260928 02:58:37.891902 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "fffdab5d22c64585ac469a30c985b66c"
format_stamp: "Formatted at 2026-09-28 02:58:37 on dist-test-slave-k5rg"
server_key: "6c8c957342ee140facf1fe7c27cb2d1e"
server_key_iv: "4e3b3bef24894a4c2fde6a7ea140ab8b"
server_key_version: "encryptionkey@0"
I20260928 02:58:37.896565 15795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:58:37.899822 15810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:37.901074 15795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:58:37.901212 15795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "fffdab5d22c64585ac469a30c985b66c"
format_stamp: "Formatted at 2026-09-28 02:58:37 on dist-test-slave-k5rg"
server_key: "6c8c957342ee140facf1fe7c27cb2d1e"
server_key_iv: "4e3b3bef24894a4c2fde6a7ea140ab8b"
server_key_version: "encryptionkey@0"
I20260928 02:58:37.901373 15795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:37.914981 15795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:37.918684 15795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:37.918926 15795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:37.919754 15795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:37.920930 15795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:37.921010 15795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:37.921119 15795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:37.921172 15795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:37.951079 15795 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:40701
I20260928 02:58:37.951114 15922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:40701 every 8 connection(s)
I20260928 02:58:37.952282 15795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:58:37.956398 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 15795
I20260928 02:58:37.956605 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:58:37.957108 13017 external_mini_cluster.cc:1428] Setting key 46a6bf5968c43e2586dbd4560de10734
Sep 28 02:58:37 dist-test-slave-k5rg krb5kdc[15422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:37.971179 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:37.954747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:34189 (local address 127.12.182.126:38475)
0928 02:58:37.954943 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:37.954949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:37.956069 (+ 1120us) server_negotiation.cc:407] Connection header received
0928 02:58:37.958301 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:37.958306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:37.958388 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:37.958470 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:37.959895 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.960611 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.962140 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.962522 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.962623 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:37.965997 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:37.966002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:37.966005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:37.966036 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:37.968344 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:37.969039 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:37.969043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:37.969045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:37.969160 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:37.969550 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:37.969553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:37.969555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:37.969848 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:37.969989 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:37.970626 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 02:58:37.970835 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:58:37.972131 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:37.955238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:38475 (local address 127.12.182.67:34189)
0928 02:58:37.955900 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:37.955941 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:37.958050 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:37.958698 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:37.958706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:37.959198 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:37.959740 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:37.959763 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.960739 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.960742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:37.962015 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:37.962028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:37.962721 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:37.962728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:37.962965 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:37.963630 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:37.963651 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:37.965764 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:37.968496 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:37.968505 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:37.968520 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:37.968923 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:37.969256 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:37.969261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:37.969265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:37.969441 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:37.969971 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:37.969978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:37.970276 (+ 298us) client_negotiation.cc:769] Sending connection context
0928 02:58:37.970493 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 02:58:37.970989 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":249,"threads_started":1}
I20260928 02:58:37.973363 15923 heartbeater.cc:347] Connected to a master server at 127.12.182.126:38475
I20260928 02:58:37.973628 15923 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:37.974272 15923 heartbeater.cc:511] Master 127.12.182.126:38475 requested a full tablet report, sending...
I20260928 02:58:37.975479 15467 ts_manager.cc:194] Registered new tserver with Master: fffdab5d22c64585ac469a30c985b66c (127.12.182.67:40701)
I20260928 02:58:37.976249 15467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:34189
I20260928 02:58:37.983606 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:58:38.009797 15658 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:38.007468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.12.182.126:38475)
0928 02:58:38.007694 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:38.007701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:38.008012 (+ 311us) server_negotiation.cc:407] Connection header received
0928 02:58:38.008150 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:38.008154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:38.008228 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:38.008331 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:38.009551 (+ 1220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47992: BlockingRecv error: recv got EOF from 127.0.0.1:47992 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:58:38.013399 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 15528
W20260928 02:58:38.015681 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 15528: Not authorized: Client connection negotiation failed: client connection to 127.12.182.65:45053: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:58:38.015853 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 15528
I20260928 02:58:38.028579 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 15662
W20260928 02:58:38.030897 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 15662: Not authorized: Client connection negotiation failed: client connection to 127.12.182.66:33245: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:58:38.031045 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 15662
I20260928 02:58:38.042472 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 15795
W20260928 02:58:38.045068 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 15795: Not authorized: Client connection negotiation failed: client connection to 127.12.182.67:40701: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:58:38.045177 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 15795
I20260928 02:58:38.056592 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 15435
I20260928 02:58:38.057698 15497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:38.158308 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 15435
2026-09-28T02:58:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3254 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:38 dist-test-slave-k5rg krb5kdc[15940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:38 dist-test-slave-k5rg krb5kdc[15940](info): set up 2 sockets
Sep 28 02:58:38 dist-test-slave-k5rg krb5kdc[15940](info): commencing operation
krb5kdc: starting...
W20260928 02:58:39.141776 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:58:39 dist-test-slave-k5rg krb5kdc[15940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:58:39.305910 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:34149
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:34149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:39.606053 15953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:39.606318 15953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:39.606439 15953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:39.613466 15953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:39.613561 15953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:39.613641 15953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:39.613716 15953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:39.613782 15953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:39.623402 15953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:34149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:34149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:39.624672 15953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:39.626000 15953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:39.636826 15958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:39.636761 15959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:39.636797 15961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:39.637475 15953 server_base.cc:1034] running on GCE node
I20260928 02:58:39.638211 15953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:39.639519 15953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:39.640723 15953 hybrid_clock.cc:648] HybridClock initialized: now 1790564319640669 us; error 46 us; skew 500 ppm
I20260928 02:58:39.641028 15953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:39 dist-test-slave-k5rg krb5kdc[15940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564319, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:39.643850 15953 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:58:39.645107 15953 webserver.cc:460] Webserver started at http://127.12.182.126:33023/ using document root <none> and password file <none>
I20260928 02:58:39.645771 15953 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:39.645888 15953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:39.646170 15953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:39.648873 15953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "5c2a67b2697944b1b380cf4231937bb5"
format_stamp: "Formatted at 2026-09-28 02:58:39 on dist-test-slave-k5rg"
server_key: "e8aaf9457c739b36f4afaf11165f9b4a"
server_key_iv: "b2d032d14f16489673ee729180d29714"
server_key_version: "encryptionkey@0"
I20260928 02:58:39.649596 15953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "5c2a67b2697944b1b380cf4231937bb5"
format_stamp: "Formatted at 2026-09-28 02:58:39 on dist-test-slave-k5rg"
server_key: "e8aaf9457c739b36f4afaf11165f9b4a"
server_key_iv: "b2d032d14f16489673ee729180d29714"
server_key_version: "encryptionkey@0"
I20260928 02:58:39.654258 15953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:58:39.657243 15968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:39.658463 15953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:58:39.658613 15953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "5c2a67b2697944b1b380cf4231937bb5"
format_stamp: "Formatted at 2026-09-28 02:58:39 on dist-test-slave-k5rg"
server_key: "e8aaf9457c739b36f4afaf11165f9b4a"
server_key_iv: "b2d032d14f16489673ee729180d29714"
server_key_version: "encryptionkey@0"
I20260928 02:58:39.658780 15953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:39.687109 15953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:39.691073 15953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:39.691310 15953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:39.707470 15953 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:34149
I20260928 02:58:39.707509 16019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:34149 every 8 connection(s)
I20260928 02:58:39.708676 15953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:39.710033 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 15953
I20260928 02:58:39.710258 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:39.711042 13017 external_mini_cluster.cc:1428] Setting key c280d36f5659b11cde85853b3c75b160
I20260928 02:58:39.712968 16020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:58:39 dist-test-slave-k5rg krb5kdc[15940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:39.725574 16020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5: Bootstrap starting.
I20260928 02:58:39.728842 16020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:39.730110 16020 log.cc:824] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:39.732802 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:39.712820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.12.182.126:34149)
0928 02:58:39.713595 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:39.713634 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:39.713679 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:58:39.715281 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:39.715303 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:39.715682 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:39.716012 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:39.717722 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:39.719032 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:39.720720 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:39.721344 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:39.721496 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:39.725969 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:39.725978 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:39.725995 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:39.726029 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:39.728796 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:39.729534 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:39.729540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:39.729546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:39.729722 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:39.730132 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:39.730137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:39.730139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:39.730839 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:39.731133 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:39.731500 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:58:39.731862 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":356,"threads_started":1}
I20260928 02:58:39.732945 16020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5: No bootstrap required, opened a new log
I20260928 02:58:39.736395 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:58:39.742244 16020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2a67b2697944b1b380cf4231937bb5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34149 } }
I20260928 02:58:39.742808 16020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:39.742908 16020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c2a67b2697944b1b380cf4231937bb5, State: Initialized, Role: FOLLOWER
I20260928 02:58:39.743556 16020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2a67b2697944b1b380cf4231937bb5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34149 } }
I20260928 02:58:39.743818 16020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:39.743964 16020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:39.744133 16020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:39.747449 16020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2a67b2697944b1b380cf4231937bb5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34149 } }
I20260928 02:58:39.748036 16020 leader_election.cc:302] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c2a67b2697944b1b380cf4231937bb5; no voters:
I20260928 02:58:39.749075 16020 leader_election.cc:288] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:39.749159 16026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:39.754446 16020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:39.756212 16026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [term 1 LEADER]: Becoming Leader. State: Replica: 5c2a67b2697944b1b380cf4231937bb5, State: Running, Role: LEADER
I20260928 02:58:39.756908 16026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2a67b2697944b1b380cf4231937bb5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34149 } }
I20260928 02:58:39.762796 16028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c2a67b2697944b1b380cf4231937bb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2a67b2697944b1b380cf4231937bb5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34149 } } }
I20260928 02:58:39.763190 16028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:39.763721 16027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c2a67b2697944b1b380cf4231937bb5. Latest consensus state: current_term: 1 leader_uuid: "5c2a67b2697944b1b380cf4231937bb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2a67b2697944b1b380cf4231937bb5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34149 } } }
I20260928 02:58:39.763996 16027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:39.773373 16035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:39.783092 16035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:39.800316 16035 catalog_manager.cc:1347] Generated new cluster ID: 209c7878379f4b39b8a48ddf673e8801
I20260928 02:58:39.800622 16035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:39.811847 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:39.800492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.12.182.126:34149)
0928 02:58:39.800667 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:39.800673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:39.800695 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:58:39.800763 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:39.800768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:39.800851 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:39.800951 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:39.802455 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:39.803608 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:39.806578 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:39.807232 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:39.807351 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:39.810128 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:39.810135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:39.810140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:39.810950 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:39.811005 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:39.811040 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:39.811128 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:39.811351 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:58:39.811450 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:58:39.824980 16035 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:39.826716 16035 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:39.838636 16035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c2a67b2697944b1b380cf4231937bb5: Generated new TSK 0
I20260928 02:58:39.839380 16035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:58:39.848796 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 15953
I20260928 02:58:39.850003 16015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:39.938098 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 15953
2026-09-28T02:58:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1774 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:40 dist-test-slave-k5rg krb5kdc[16055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:40 dist-test-slave-k5rg krb5kdc[16055](info): set up 2 sockets
Sep 28 02:58:40 dist-test-slave-k5rg krb5kdc[16055](info): commencing operation
krb5kdc: starting...
W20260928 02:58:40.918182 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:58:41 dist-test-slave-k5rg krb5kdc[16055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:58:41.092648 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:40023
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:41.402369 16068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:41.402643 16068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:41.402773 16068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:41.409916 16068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:41.410017 16068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:41.410095 16068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:41.410167 16068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:41.410228 16068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:41.419872 16068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:40023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:40023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:41.421106 16068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:41.422348 16068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:41.433349 16073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:41.433403 16076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:41.433230 16074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:41.434404 16068 server_base.cc:1034] running on GCE node
I20260928 02:58:41.435205 16068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:41.436547 16068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:41.437709 16068 hybrid_clock.cc:648] HybridClock initialized: now 1790564321437662 us; error 31 us; skew 500 ppm
I20260928 02:58:41.438035 16068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:41 dist-test-slave-k5rg krb5kdc[16055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564321, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:41.444072 16068 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:58:41.445454 16068 webserver.cc:460] Webserver started at http://127.12.182.126:41759/ using document root <none> and password file <none>
I20260928 02:58:41.446118 16068 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:41.446234 16068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:41.446485 16068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:41.449666 16068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "50205809318346a6a0142562bc2090bf"
format_stamp: "Formatted at 2026-09-28 02:58:41 on dist-test-slave-k5rg"
server_key: "af8af3f0cf4b61af6f853922a5eed70e"
server_key_iv: "b65ff611372d348fce73065fdba876de"
server_key_version: "encryptionkey@0"
I20260928 02:58:41.450441 16068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "50205809318346a6a0142562bc2090bf"
format_stamp: "Formatted at 2026-09-28 02:58:41 on dist-test-slave-k5rg"
server_key: "af8af3f0cf4b61af6f853922a5eed70e"
server_key_iv: "b65ff611372d348fce73065fdba876de"
server_key_version: "encryptionkey@0"
I20260928 02:58:41.455364 16068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:58:41.458670 16083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:41.459995 16068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:58:41.460150 16068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "50205809318346a6a0142562bc2090bf"
format_stamp: "Formatted at 2026-09-28 02:58:41 on dist-test-slave-k5rg"
server_key: "af8af3f0cf4b61af6f853922a5eed70e"
server_key_iv: "b65ff611372d348fce73065fdba876de"
server_key_version: "encryptionkey@0"
I20260928 02:58:41.460317 16068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:41.473711 16068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:41.477641 16068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:41.477901 16068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:41.493534 16068 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:40023
I20260928 02:58:41.493558 16134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:40023 every 8 connection(s)
I20260928 02:58:41.494789 16068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:41.497663 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 16068
I20260928 02:58:41.497869 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:41.498613 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:41.498646 13017 external_mini_cluster.cc:1428] Setting key 85a0d9dae5614b8545af13088fc4fd24
I20260928 02:58:41.512105 16135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf: Bootstrap starting.
Sep 28 02:58:41 dist-test-slave-k5rg krb5kdc[16055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:41.515604 16135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:41.516830 16135 log.cc:824] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:41.520036 16135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf: No bootstrap required, opened a new log
I20260928 02:58:41.521234 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:41.500639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.12.182.126:40023)
0928 02:58:41.501364 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:41.501375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:41.501421 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:58:41.503153 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:41.503173 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:41.503572 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:41.503897 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:41.505535 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:41.506787 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:41.508401 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:41.509110 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:41.509231 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:41.513506 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:41.513516 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:41.513534 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:41.513586 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:41.517114 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:41.517866 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:41.517872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:41.517877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:41.518029 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:41.518392 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:41.518397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:41.518400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:41.519020 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:41.519310 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:41.519614 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 02:58:41.519996 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":323,"threads_started":1}
I20260928 02:58:41.527951 16135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50205809318346a6a0142562bc2090bf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40023 } }
I20260928 02:58:41.528265 16135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:41.528357 16135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50205809318346a6a0142562bc2090bf, State: Initialized, Role: FOLLOWER
I20260928 02:58:41.528853 16135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50205809318346a6a0142562bc2090bf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40023 } }
I20260928 02:58:41.529045 16135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:41.529137 16135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:41.529255 16135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:41.531241 16135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50205809318346a6a0142562bc2090bf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40023 } }
I20260928 02:58:41.531667 16135 leader_election.cc:302] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50205809318346a6a0142562bc2090bf; no voters:
I20260928 02:58:41.532186 16135 leader_election.cc:288] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:41.532315 16140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:41.533468 16140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [term 1 LEADER]: Becoming Leader. State: Replica: 50205809318346a6a0142562bc2090bf, State: Running, Role: LEADER
I20260928 02:58:41.533962 16140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50205809318346a6a0142562bc2090bf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40023 } }
I20260928 02:58:41.534373 16135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:41.536165 16141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50205809318346a6a0142562bc2090bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50205809318346a6a0142562bc2090bf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40023 } } }
I20260928 02:58:41.536440 16141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:41.537240 16146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:41.537089 16142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50205809318346a6a0142562bc2090bf. Latest consensus state: current_term: 1 leader_uuid: "50205809318346a6a0142562bc2090bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50205809318346a6a0142562bc2090bf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40023 } } }
I20260928 02:58:41.537948 16142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:41.543097 16146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:41.551457 16146 catalog_manager.cc:1347] Generated new cluster ID: 49d0ac001b7240b2a36c4828a321c1b2
I20260928 02:58:41.551620 16146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:41.564419 16146 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:41.566308 16146 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:41.578528 16146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50205809318346a6a0142562bc2090bf: Generated new TSK 0
I20260928 02:58:41.579311 16146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:58:41.633056 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40023
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:41.960719 16161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:41.960990 16161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:41.961090 16161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:41.968274 16161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:41.968374 16161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:41.968492 16161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:58:41.978093 16161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:41.979378 16161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:41.980647 16161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:41.992856 16166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:41.992996 16167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:41.992828 16169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:41.993700 16161 server_base.cc:1034] running on GCE node
I20260928 02:58:41.994234 16161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:41.995074 16161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:41.996265 16161 hybrid_clock.cc:648] HybridClock initialized: now 1790564321996222 us; error 30 us; skew 500 ppm
I20260928 02:58:41.996547 16161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:41 dist-test-slave-k5rg krb5kdc[16055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564321, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:41.999579 16161 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:58:42.001116 16161 webserver.cc:460] Webserver started at http://127.12.182.65:42647/ using document root <none> and password file <none>
I20260928 02:58:42.001791 16161 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:42.001914 16161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:42.002190 16161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:42.004972 16161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "c956db8be3474f3b8a7a560d8945b631"
format_stamp: "Formatted at 2026-09-28 02:58:41 on dist-test-slave-k5rg"
server_key: "0023fe2637af0429dc46e59948d43ac4"
server_key_iv: "0337d9d1b650193016ceddc6fb57445b"
server_key_version: "encryptionkey@0"
I20260928 02:58:42.005738 16161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "c956db8be3474f3b8a7a560d8945b631"
format_stamp: "Formatted at 2026-09-28 02:58:41 on dist-test-slave-k5rg"
server_key: "0023fe2637af0429dc46e59948d43ac4"
server_key_iv: "0337d9d1b650193016ceddc6fb57445b"
server_key_version: "encryptionkey@0"
I20260928 02:58:42.010442 16161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:58:42.013574 16176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:42.014811 16161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 02:58:42.014956 16161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "c956db8be3474f3b8a7a560d8945b631"
format_stamp: "Formatted at 2026-09-28 02:58:41 on dist-test-slave-k5rg"
server_key: "0023fe2637af0429dc46e59948d43ac4"
server_key_iv: "0337d9d1b650193016ceddc6fb57445b"
server_key_version: "encryptionkey@0"
I20260928 02:58:42.015129 16161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:42.031399 16161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:42.035125 16161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:42.035449 16161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:42.036363 16161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:42.037657 16161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:42.037734 16161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:42.037858 16161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:42.037914 16161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:42.067435 16161 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:41303
I20260928 02:58:42.067476 16288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:41303 every 8 connection(s)
I20260928 02:58:42.068651 16161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:58:42.071269 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 16161
I20260928 02:58:42.071463 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:58:42.072274 13017 external_mini_cluster.cc:1428] Setting key 2a09d40c1d852e03f66ccfb362fe10ee
Sep 28 02:58:42 dist-test-slave-k5rg krb5kdc[16055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564321, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:42.088052 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:42.070870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:41523 (local address 127.12.182.126:40023)
0928 02:58:42.071274 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:42.071279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:42.072340 (+ 1061us) server_negotiation.cc:407] Connection header received
0928 02:58:42.074149 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:42.074154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:42.074234 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:42.074324 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:42.075757 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.076477 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.078033 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.078545 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.078655 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:42.082488 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:42.082493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:42.082496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:42.082530 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:42.084906 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:42.085814 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:42.085818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:42.085821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:42.085937 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:42.086391 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:42.086394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:42.086396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:42.086643 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:42.086745 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:42.087618 (+ 873us) server_negotiation.cc:299] Negotiation successful
0928 02:58:42.087782 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":211,"threads_started":1}
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
I20260928 02:58:42.089296 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:42.071477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40023 (local address 127.12.182.65:41523)
0928 02:58:42.072201 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:42.072236 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:42.073898 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:42.074560 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:42.074568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:42.075060 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:42.075594 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:42.075612 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.076628 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.076632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:42.077897 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:42.077910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.078762 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.078781 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:42.079036 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:42.079655 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:42.079688 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:42.082274 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:42.085044 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:42.085052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:42.085070 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:42.085702 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:42.086041 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:42.086047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:42.086051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:42.086297 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:42.086756 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:42.086765 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:42.087222 (+ 457us) client_negotiation.cc:769] Sending connection context
0928 02:58:42.087530 (+ 308us) client_negotiation.cc:240] Negotiation successful
0928 02:58:42.088082 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":288,"threads_started":1}
I20260928 02:58:42.090878 16289 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
I20260928 02:58:42.091195 16289 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:42.091974 16289 heartbeater.cc:511] Master 127.12.182.126:40023 requested a full tablet report, sending...
I20260928 02:58:42.094072 16100 ts_manager.cc:194] Registered new tserver with Master: c956db8be3474f3b8a7a560d8945b631 (127.12.182.65:41303)
I20260928 02:58:42.095464 16100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:41523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:58:42.110777 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40023
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:42.418166 16295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:42.418427 16295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:42.418565 16295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:42.425824 16295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:42.425925 16295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:42.426064 16295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:58:42.435771 16295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:42.437007 16295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:42.438272 16295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:42.450361 16300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:42.450434 16301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:42.450547 16303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:42.451432 16295 server_base.cc:1034] running on GCE node
I20260928 02:58:42.451988 16295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:42.452843 16295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:42.454030 16295 hybrid_clock.cc:648] HybridClock initialized: now 1790564322453972 us; error 49 us; skew 500 ppm
I20260928 02:58:42.454315 16295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:42 dist-test-slave-k5rg krb5kdc[16055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:42.457324 16295 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:58:42.458653 16295 webserver.cc:460] Webserver started at http://127.12.182.66:34731/ using document root <none> and password file <none>
I20260928 02:58:42.459270 16295 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:42.459384 16295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:42.459652 16295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:42.462366 16295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "c43dda8294d04f7d88c783cacfbf492a"
format_stamp: "Formatted at 2026-09-28 02:58:42 on dist-test-slave-k5rg"
server_key: "4ad9ec274282473f2c8c9bc64fef8c7f"
server_key_iv: "59e1c73c21ad66fb9c6862b196e24f57"
server_key_version: "encryptionkey@0"
I20260928 02:58:42.463086 16295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "c43dda8294d04f7d88c783cacfbf492a"
format_stamp: "Formatted at 2026-09-28 02:58:42 on dist-test-slave-k5rg"
server_key: "4ad9ec274282473f2c8c9bc64fef8c7f"
server_key_iv: "59e1c73c21ad66fb9c6862b196e24f57"
server_key_version: "encryptionkey@0"
I20260928 02:58:42.467911 16295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 02:58:42.471086 16310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:42.472332 16295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:58:42.472471 16295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "c43dda8294d04f7d88c783cacfbf492a"
format_stamp: "Formatted at 2026-09-28 02:58:42 on dist-test-slave-k5rg"
server_key: "4ad9ec274282473f2c8c9bc64fef8c7f"
server_key_iv: "59e1c73c21ad66fb9c6862b196e24f57"
server_key_version: "encryptionkey@0"
I20260928 02:58:42.472641 16295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:42.486672 16295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:42.490494 16295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:42.490753 16295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:42.491626 16295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:42.492867 16295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:42.492941 16295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:42.493070 16295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:42.493125 16295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:42.523336 16295 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:42067
I20260928 02:58:42.523373 16422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:42067 every 8 connection(s)
I20260928 02:58:42.524519 16295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:58:42.526417 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 16295
I20260928 02:58:42.526564 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:58:42.527037 13017 external_mini_cluster.cc:1428] Setting key 60f3c60d68a86d1506a6b1ec65c5a655
Sep 28 02:58:42 dist-test-slave-k5rg krb5kdc[16055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
I20260928 02:58:42.544659 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:42.527047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:46337 (local address 127.12.182.126:40023)
0928 02:58:42.527244 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:42.527250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:42.528393 (+ 1143us) server_negotiation.cc:407] Connection header received
0928 02:58:42.530211 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:42.530218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:42.530298 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:42.530409 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:42.531793 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.532727 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.534299 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.534892 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.535027 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:42.538382 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:42.538388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:42.538393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:42.538433 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:42.541672 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:42.542397 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:42.542403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:42.542406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:42.542570 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:42.542945 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:42.542950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:42.542953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:42.543297 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:42.543451 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:42.544108 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 02:58:42.544336 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:58:42.545703 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:42.527476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40023 (local address 127.12.182.66:46337)
0928 02:58:42.528228 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:42.528276 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:42.529950 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:42.530595 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:42.530603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:42.531086 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:42.531643 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:42.531660 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.532846 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.532850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:42.534166 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:42.534179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.535096 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.535101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:42.535286 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:42.535946 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:42.535967 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:42.538156 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:42.541811 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:42.541819 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:42.541833 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:42.542270 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:42.542660 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:42.542663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:42.542665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:42.542838 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:42.543403 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:42.543408 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:42.543727 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 02:58:42.543966 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 02:58:42.544485 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":312,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
I20260928 02:58:42.547102 16423 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40023
I20260928 02:58:42.547385 16423 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:42.548076 16423 heartbeater.cc:511] Master 127.12.182.126:40023 requested a full tablet report, sending...
I20260928 02:58:42.549453 16100 ts_manager.cc:194] Registered new tserver with Master: c43dda8294d04f7d88c783cacfbf492a (127.12.182.66:42067)
I20260928 02:58:42.550287 16100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:46337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:58:42.566854 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40023
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:42.873049 16428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:42.873284 16428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:42.873407 16428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:42.880322 16428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:42.880419 16428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:42.880571 16428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:58:42.889737 16428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:42.890865 16428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:42.892096 16428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:42.904551 16433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:42.904520 16436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:42.904481 16434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:42.905248 16428 server_base.cc:1034] running on GCE node
I20260928 02:58:42.905835 16428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:42.906715 16428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:42.907951 16428 hybrid_clock.cc:648] HybridClock initialized: now 1790564322907888 us; error 52 us; skew 500 ppm
I20260928 02:58:42.908234 16428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:42 dist-test-slave-k5rg krb5kdc[16055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:42.911163 16428 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:58:42.912453 16428 webserver.cc:460] Webserver started at http://127.12.182.67:37467/ using document root <none> and password file <none>
I20260928 02:58:42.913071 16428 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:42.913177 16428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:42.913442 16428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:42.916139 16428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "f35bc6c29c3c45eb951937958c8d7e72"
format_stamp: "Formatted at 2026-09-28 02:58:42 on dist-test-slave-k5rg"
server_key: "f4d924373113b23a594bf071e46bb037"
server_key_iv: "a9b45ae1c889375e4ebcafb14fe18d8f"
server_key_version: "encryptionkey@0"
I20260928 02:58:42.916828 16428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "f35bc6c29c3c45eb951937958c8d7e72"
format_stamp: "Formatted at 2026-09-28 02:58:42 on dist-test-slave-k5rg"
server_key: "f4d924373113b23a594bf071e46bb037"
server_key_iv: "a9b45ae1c889375e4ebcafb14fe18d8f"
server_key_version: "encryptionkey@0"
I20260928 02:58:42.921627 16428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 02:58:42.924808 16443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:42.926061 16428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 02:58:42.926200 16428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "f35bc6c29c3c45eb951937958c8d7e72"
format_stamp: "Formatted at 2026-09-28 02:58:42 on dist-test-slave-k5rg"
server_key: "f4d924373113b23a594bf071e46bb037"
server_key_iv: "a9b45ae1c889375e4ebcafb14fe18d8f"
server_key_version: "encryptionkey@0"
I20260928 02:58:42.926362 16428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:42.943601 16428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:42.947527 16428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:42.947795 16428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:42.948648 16428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:42.949919 16428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:42.949993 16428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:42.950112 16428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:42.950165 16428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:42.978737 16428 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:34753
I20260928 02:58:42.978767 16555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:34753 every 8 connection(s)
I20260928 02:58:42.979923 16428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:58:42.982952 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 16428
I20260928 02:58:42.983106 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:58:42.983542 13017 external_mini_cluster.cc:1428] Setting key def30e1d1b3998107361da5bce419a1d
Sep 28 02:58:42 dist-test-slave-k5rg krb5kdc[16055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:42.998633 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:42.982280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:57491 (local address 127.12.182.126:40023)
0928 02:58:42.982434 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:42.982439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:42.983642 (+ 1203us) server_negotiation.cc:407] Connection header received
0928 02:58:42.985572 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:42.985576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:42.985684 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:42.985762 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:42.987189 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.987895 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.989334 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.989774 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.989863 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:42.993064 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:42.993069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:42.993072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:42.993107 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:42.995641 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:42.996334 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:42.996337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:42.996340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:42.996467 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:42.996857 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:42.996860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:42.996863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:42.997161 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:42.997296 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:42.998077 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 02:58:42.998290 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:58:42.999585 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:42.982735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40023 (local address 127.12.182.67:57491)
0928 02:58:42.983486 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:42.983542 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:42.985306 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:42.986038 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:42.986049 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:42.986523 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:42.987049 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:42.987066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.988029 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.988034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:42.989214 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:42.989226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:42.989983 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:42.989989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:42.990185 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:42.990745 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:42.990771 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:42.992844 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:42.995768 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:42.995775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:42.995787 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:42.996217 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:42.996564 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:42.996568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:42.996570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:42.996745 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:42.997299 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:42.997308 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:42.997736 (+ 428us) client_negotiation.cc:769] Sending connection context
0928 02:58:42.998006 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 02:58:42.998469 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":303,"threads_started":1}
I20260928 02:58:43.000844 16556 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40023
I20260928 02:58:43.001111 16556 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:43.001823 16556 heartbeater.cc:511] Master 127.12.182.126:40023 requested a full tablet report, sending...
I20260928 02:58:43.003127 16100 ts_manager.cc:194] Registered new tserver with Master: f35bc6c29c3c45eb951937958c8d7e72 (127.12.182.67:34753)
I20260928 02:58:43.003931 16100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:57491
I20260928 02:58:43.010778 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:58:43 dist-test-slave-k5rg krb5kdc[16055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564323, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:58:43 dist-test-slave-k5rg krb5kdc[16055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564323, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:43.053807 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:43.040847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.12.182.126:40023)
0928 02:58:43.041009 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:43.041017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:43.041439 (+ 422us) server_negotiation.cc:407] Connection header received
0928 02:58:43.041637 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:43.041643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:43.041713 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:43.041798 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:43.043102 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:43.043829 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:43.045032 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:43.045441 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:43.045559 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:43.049056 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:43.049061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:43.049065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:43.049098 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:43.051345 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:43.051937 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:43.051941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:43.051943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:43.052069 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:43.052429 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:43.052433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:43.052435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:43.052758 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:43.052977 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:43.053260 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 02:58:43.053474 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 02:58:43.054332 16100 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37102
I20260928 02:58:43.055871 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16161
Sep 28 02:58:43 dist-test-slave-k5rg krb5kdc[16055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564323, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:58:43.071233 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:43.057250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.12.182.65:41303)
0928 02:58:43.057780 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:43.057786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:43.057809 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:58:43.057868 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:43.057874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:43.058077 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:43.058199 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:43.059145 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:43.060154 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:43.061356 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:43.061970 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:43.062095 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:43.065724 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:43.065735 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:43.065748 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:43.065779 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:43.068560 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:43.069160 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:43.069168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:43.069170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:43.069284 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:43.069663 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:43.069668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:43.069670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:43.069999 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:43.070278 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:43.070562 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:58:43.070821 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":279,"threads_started":1}
W20260928 02:58:43.072381 16284 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47274
W20260928 02:58:43.073344 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16161: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:58:43.073542 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 16161
I20260928 02:58:43.085979 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16295
Sep 28 02:58:43 dist-test-slave-k5rg krb5kdc[16055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564323, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 02:58:43.100797 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:43.086744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.12.182.66:42067)
0928 02:58:43.087298 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:43.087306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:43.087330 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:58:43.087392 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:43.087398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:43.087603 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:43.087730 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:43.088864 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:43.089936 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:43.091156 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:43.091719 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:43.091820 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:43.095690 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:43.095702 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:43.095717 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:43.095752 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:43.098322 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:43.098942 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:43.098949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:43.098952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:43.099070 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:43.099427 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:43.099431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:43.099433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:43.099762 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:43.099984 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:43.100241 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:58:43.100435 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":282,"threads_started":1}
W20260928 02:58:43.102082 16418 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42240
W20260928 02:58:43.103111 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16295: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:58:43.103242 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 16295
I20260928 02:58:43.115363 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16428
Sep 28 02:58:43 dist-test-slave-k5rg krb5kdc[16055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564323, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 02:58:43.131371 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:43.116305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.12.182.67:34753)
0928 02:58:43.116911 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:43.116918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:43.116943 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:58:43.117003 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:43.117009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:43.117218 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:43.117361 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:43.118661 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:43.119689 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:43.121029 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:43.121580 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:43.121717 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:43.125748 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:43.125760 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:43.125776 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:43.125815 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:43.128447 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:43.129129 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:43.129136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:43.129143 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:43.129271 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:43.129795 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:43.129798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:43.129801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:43.130179 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:43.130470 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:43.130761 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 02:58:43.131030 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":323,"threads_started":1}
W20260928 02:58:43.132735 16551 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35972
W20260928 02:58:43.133823 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16428: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:58:43.134013 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 16428
I20260928 02:58:43.145958 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16068
I20260928 02:58:43.147040 16130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:43.249155 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 16068
2026-09-28T02:58:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3315 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:43 dist-test-slave-k5rg krb5kdc[16573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:43 dist-test-slave-k5rg krb5kdc[16573](info): set up 2 sockets
Sep 28 02:58:43 dist-test-slave-k5rg krb5kdc[16573](info): commencing operation
krb5kdc: starting...
W20260928 02:58:44.246043 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:58:44 dist-test-slave-k5rg krb5kdc[16573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:58:44.418991 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41225
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:41225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:44.715915 16586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:44.716194 16586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:44.716331 16586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:44.723793 16586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:44.723917 16586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:44.723994 16586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:44.724074 16586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:44.724133 16586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:44.733491 16586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:41225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:44.734699 16586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:44.735932 16586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:44.746667 16592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:44.746717 16594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:44.746795 16591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:44.747524 16586 server_base.cc:1034] running on GCE node
I20260928 02:58:44.748217 16586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:44.749467 16586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:44.750667 16586 hybrid_clock.cc:648] HybridClock initialized: now 1790564324750605 us; error 41 us; skew 500 ppm
I20260928 02:58:44.750964 16586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:44 dist-test-slave-k5rg krb5kdc[16573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564324, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:44.753854 16586 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:58:44.755102 16586 webserver.cc:460] Webserver started at http://127.12.182.126:32893/ using document root <none> and password file <none>
I20260928 02:58:44.755692 16586 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:44.755782 16586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:44.756021 16586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:44.758741 16586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "94af14f6f8734a9a85a65971d4638f3f"
format_stamp: "Formatted at 2026-09-28 02:58:44 on dist-test-slave-k5rg"
server_key: "ed8d3f92631043cf37b8e23bbc3ea536"
server_key_iv: "eac2d621a801303c3aa2f8363c1f3349"
server_key_version: "encryptionkey@0"
I20260928 02:58:44.759413 16586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "94af14f6f8734a9a85a65971d4638f3f"
format_stamp: "Formatted at 2026-09-28 02:58:44 on dist-test-slave-k5rg"
server_key: "ed8d3f92631043cf37b8e23bbc3ea536"
server_key_iv: "eac2d621a801303c3aa2f8363c1f3349"
server_key_version: "encryptionkey@0"
I20260928 02:58:44.763980 16586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:58:44.767094 16601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:44.768270 16586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:58:44.768405 16586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "94af14f6f8734a9a85a65971d4638f3f"
format_stamp: "Formatted at 2026-09-28 02:58:44 on dist-test-slave-k5rg"
server_key: "ed8d3f92631043cf37b8e23bbc3ea536"
server_key_iv: "eac2d621a801303c3aa2f8363c1f3349"
server_key_version: "encryptionkey@0"
I20260928 02:58:44.768565 16586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:44.791597 16586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:44.796629 16586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:44.796893 16586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:44.814122 16586 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:41225
I20260928 02:58:44.814152 16652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:41225 every 8 connection(s)
I20260928 02:58:44.815371 16586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:44.818893 16653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:44.822710 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 16586
I20260928 02:58:44.822932 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:44.823709 13017 external_mini_cluster.cc:1428] Setting key c7a715b8493a69e51d92c81196148f1c
I20260928 02:58:44.829782 16653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f: Bootstrap starting.
I20260928 02:58:44.832605 16653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:44.833506 16653 log.cc:824] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:44.836133 16653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f: No bootstrap required, opened a new log
Sep 28 02:58:44 dist-test-slave-k5rg krb5kdc[16573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:44.844010 16653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af14f6f8734a9a85a65971d4638f3f" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41225 } }
I20260928 02:58:44.844336 16653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:44.844440 16653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af14f6f8734a9a85a65971d4638f3f, State: Initialized, Role: FOLLOWER
I20260928 02:58:44.844842 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:44.825809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.12.182.126:41225)
0928 02:58:44.826509 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:44.826520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:44.826565 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:58:44.828600 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:44.828622 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:44.828971 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:44.829298 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:44.830757 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:44.832043 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:44.833294 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:44.834050 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:44.834249 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:44.838034 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:44.838042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:44.838055 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:44.838091 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:44.841002 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:44.841679 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:44.841685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:44.841690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:44.841855 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:44.842219 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:44.842223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:44.842225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:44.842751 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:44.843036 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:44.843452 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 02:58:44.843822 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":318,"threads_started":1}
I20260928 02:58:44.845023 16653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af14f6f8734a9a85a65971d4638f3f" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41225 } }
I20260928 02:58:44.845304 16653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:44.845445 16653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:44.845651 16653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:44.848145 16653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af14f6f8734a9a85a65971d4638f3f" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41225 } }
I20260928 02:58:44.848549 16653 leader_election.cc:302] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94af14f6f8734a9a85a65971d4638f3f; no voters:
I20260928 02:58:44.849025 16653 leader_election.cc:288] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:44.849143 16658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:44.849787 16658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [term 1 LEADER]: Becoming Leader. State: Replica: 94af14f6f8734a9a85a65971d4638f3f, State: Running, Role: LEADER
I20260928 02:58:44.850281 16658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af14f6f8734a9a85a65971d4638f3f" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41225 } }
I20260928 02:58:44.850834 16653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:44.852734 16659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94af14f6f8734a9a85a65971d4638f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af14f6f8734a9a85a65971d4638f3f" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41225 } } }
I20260928 02:58:44.853082 16659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:44.854475 16660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94af14f6f8734a9a85a65971d4638f3f. Latest consensus state: current_term: 1 leader_uuid: "94af14f6f8734a9a85a65971d4638f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af14f6f8734a9a85a65971d4638f3f" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41225 } } }
I20260928 02:58:44.854692 16663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:44.854739 16660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:44.859998 16663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:44.869055 16663 catalog_manager.cc:1347] Generated new cluster ID: 1d3b2d7eeef444f4983f360b021034f3
I20260928 02:58:44.869163 16663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:44.877450 16663 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:44.879135 16663 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:44.897737 16663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94af14f6f8734a9a85a65971d4638f3f: Generated new TSK 0
I20260928 02:58:44.898509 16663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:58:44.948792 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41225
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:45.255939 16679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:45.256196 16679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:45.256325 16679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:45.263262 16679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:45.263350 16679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:45.263496 16679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:58:45.272789 16679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:45.273980 16679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:45.275204 16679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:45.287040 16687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:45.287106 16684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:45.287132 16685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:45.288007 16679 server_base.cc:1034] running on GCE node
I20260928 02:58:45.288569 16679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:45.289399 16679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:45.290588 16679 hybrid_clock.cc:648] HybridClock initialized: now 1790564325290533 us; error 44 us; skew 500 ppm
I20260928 02:58:45.290867 16679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:45 dist-test-slave-k5rg krb5kdc[16573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:45.293766 16679 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:58:45.295077 16679 webserver.cc:460] Webserver started at http://127.12.182.65:41211/ using document root <none> and password file <none>
I20260928 02:58:45.295704 16679 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:45.295814 16679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:45.296087 16679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:45.298878 16679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "64b521aca26e4f3e82a2ef551361985d"
format_stamp: "Formatted at 2026-09-28 02:58:45 on dist-test-slave-k5rg"
server_key: "8a6e5747ddbae2e0986ac7878930b5fd"
server_key_iv: "f930a37e96452a0c9c2a21085c41fd93"
server_key_version: "encryptionkey@0"
I20260928 02:58:45.299602 16679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "64b521aca26e4f3e82a2ef551361985d"
format_stamp: "Formatted at 2026-09-28 02:58:45 on dist-test-slave-k5rg"
server_key: "8a6e5747ddbae2e0986ac7878930b5fd"
server_key_iv: "f930a37e96452a0c9c2a21085c41fd93"
server_key_version: "encryptionkey@0"
I20260928 02:58:45.304481 16679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:58:45.307827 16694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:45.309043 16679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:58:45.309192 16679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "64b521aca26e4f3e82a2ef551361985d"
format_stamp: "Formatted at 2026-09-28 02:58:45 on dist-test-slave-k5rg"
server_key: "8a6e5747ddbae2e0986ac7878930b5fd"
server_key_iv: "f930a37e96452a0c9c2a21085c41fd93"
server_key_version: "encryptionkey@0"
I20260928 02:58:45.309370 16679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:45.326256 16679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:45.329833 16679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:45.330090 16679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:45.330945 16679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:45.332187 16679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:45.332257 16679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:45.332371 16679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:45.332419 16679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:45.360836 16679 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:35937
I20260928 02:58:45.360878 16806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:35937 every 8 connection(s)
I20260928 02:58:45.362088 16679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:58:45.366367 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 16679
I20260928 02:58:45.366575 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:58:45.367324 13017 external_mini_cluster.cc:1428] Setting key a0447d6df790c8cab240edada31a9fd7
Sep 28 02:58:45 dist-test-slave-k5rg krb5kdc[16573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:45.382256 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:45.364407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:39665 (local address 127.12.182.126:41225)
0928 02:58:45.364806 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:45.364810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:45.365667 (+ 857us) server_negotiation.cc:407] Connection header received
0928 02:58:45.368069 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:45.368074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:45.368159 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:45.368267 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:45.370078 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:45.371114 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:45.372717 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:45.373299 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:45.373411 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:45.376925 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:45.376930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:45.376933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:45.376963 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:45.379272 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:45.380058 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:45.380062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:45.380064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:45.380175 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:45.380583 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:45.380586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:45.380588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:45.380839 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:45.380940 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:45.381803 (+ 863us) server_negotiation.cc:299] Negotiation successful
0928 02:58:45.382000 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":214,"threads_started":1}
I20260928 02:58:45.383603 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:45.364866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41225 (local address 127.12.182.65:39665)
0928 02:58:45.365506 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:45.365556 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:45.367775 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:45.368564 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:45.368575 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:45.369286 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:45.369907 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:45.369925 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:45.371256 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:45.371261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:45.372593 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:45.372605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:45.373477 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:45.373481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:45.373716 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:45.374328 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:45.374352 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:45.376706 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:45.379406 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:45.379414 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:45.379431 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:45.379958 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:45.380265 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:45.380269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:45.380273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:45.380496 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:45.380949 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:45.380957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:45.381384 (+ 427us) client_negotiation.cc:769] Sending connection context
0928 02:58:45.381699 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 02:58:45.382232 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":247,"threads_started":1}
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
I20260928 02:58:45.385243 16807 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41225
I20260928 02:58:45.385512 16807 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:45.386224 16807 heartbeater.cc:511] Master 127.12.182.126:41225 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
I20260928 02:58:45.388159 16618 ts_manager.cc:194] Registered new tserver with Master: 64b521aca26e4f3e82a2ef551361985d (127.12.182.65:35937)
I20260928 02:58:45.389561 16618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:39665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:58:45.406692 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41225
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:45.706547 16813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:45.706812 16813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:45.706931 16813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:45.713910 16813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:45.714020 16813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:45.714165 16813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:58:45.723291 16813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:45.724500 16813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:45.725806 16813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:45.737710 16818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:45.737788 16821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:45.737728 16819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:45.739562 16813 server_base.cc:1034] running on GCE node
I20260928 02:58:45.740173 16813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:45.741081 16813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:45.742362 16813 hybrid_clock.cc:648] HybridClock initialized: now 1790564325742299 us; error 48 us; skew 500 ppm
I20260928 02:58:45.742648 16813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:45 dist-test-slave-k5rg krb5kdc[16573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:45.745723 16813 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:58:45.747116 16813 webserver.cc:460] Webserver started at http://127.12.182.66:36595/ using document root <none> and password file <none>
I20260928 02:58:45.747745 16813 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:45.747836 16813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:45.748085 16813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:45.750963 16813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "7faadb8220c643c38abdbdc809f9ec4a"
format_stamp: "Formatted at 2026-09-28 02:58:45 on dist-test-slave-k5rg"
server_key: "e78b367131950660e49db4b275ab3f4c"
server_key_iv: "f0e523b64fbdb754de4382222ab5a235"
server_key_version: "encryptionkey@0"
I20260928 02:58:45.751654 16813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "7faadb8220c643c38abdbdc809f9ec4a"
format_stamp: "Formatted at 2026-09-28 02:58:45 on dist-test-slave-k5rg"
server_key: "e78b367131950660e49db4b275ab3f4c"
server_key_iv: "f0e523b64fbdb754de4382222ab5a235"
server_key_version: "encryptionkey@0"
I20260928 02:58:45.756438 16813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:58:45.759680 16828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:45.760973 16813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:58:45.761101 16813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "7faadb8220c643c38abdbdc809f9ec4a"
format_stamp: "Formatted at 2026-09-28 02:58:45 on dist-test-slave-k5rg"
server_key: "e78b367131950660e49db4b275ab3f4c"
server_key_iv: "f0e523b64fbdb754de4382222ab5a235"
server_key_version: "encryptionkey@0"
I20260928 02:58:45.761262 16813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:45.790160 16813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:45.794123 16813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:45.794354 16813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:45.795256 16813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:45.796500 16813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:45.796569 16813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:45.796658 16813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:45.796707 16813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:45.826591 16813 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:40497
I20260928 02:58:45.826694 16940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:40497 every 8 connection(s)
I20260928 02:58:45.827750 16813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:58:45.831180 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 16813
I20260928 02:58:45.831355 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:58:45.831799 13017 external_mini_cluster.cc:1428] Setting key cda11c5b1bbf2c4aceb79e985f811566
Sep 28 02:58:45 dist-test-slave-k5rg krb5kdc[16573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:45.847681 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:45.830242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:60523 (local address 127.12.182.126:41225)
0928 02:58:45.830436 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:45.830443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:45.831642 (+ 1199us) server_negotiation.cc:407] Connection header received
0928 02:58:45.833999 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:45.834004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:45.834094 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:45.834199 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:45.835554 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:45.836484 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:45.837964 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:45.838536 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:45.838657 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:45.842405 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:45.842410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:45.842413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:45.842443 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:45.844744 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:45.845543 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:45.845548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:45.845551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:45.845707 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:45.846154 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:45.846157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:45.846159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:45.846396 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:45.846507 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:45.847296 (+ 789us) server_negotiation.cc:299] Negotiation successful
0928 02:58:45.847459 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:58:45.849097 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:45.830731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41225 (local address 127.12.182.66:60523)
0928 02:58:45.831477 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:45.831529 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:45.833729 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:45.834371 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:45.834379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:45.834854 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:45.835392 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:45.835409 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:45.836607 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:45.836610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:45.837842 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:45.837854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:45.838722 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:45.838729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:45.839024 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:45.839667 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:45.839699 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:45.842206 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:45.844881 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:45.844890 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:45.844905 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:45.845426 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:45.845805 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:45.845810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:45.845813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:45.846063 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:45.846510 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:45.846518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:45.846927 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 02:58:45.847232 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 02:58:45.847738 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":313,"threads_started":1}
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
I20260928 02:58:45.850514 16941 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41225
I20260928 02:58:45.850788 16941 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:45.851480 16941 heartbeater.cc:511] Master 127.12.182.126:41225 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
I20260928 02:58:45.852895 16618 ts_manager.cc:194] Registered new tserver with Master: 7faadb8220c643c38abdbdc809f9ec4a (127.12.182.66:40497)
I20260928 02:58:45.853947 16618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:60523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:58:45.872146 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41225
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:46.173432 16946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:46.173738 16946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:46.173887 16946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:46.180876 16946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:46.180990 16946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:46.181142 16946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:58:46.190457 16946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:46.191646 16946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:46.193041 16946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:46.205775 16951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:46.205834 16952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:46.206210 16954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:46.206560 16946 server_base.cc:1034] running on GCE node
I20260928 02:58:46.207118 16946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:46.207943 16946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:46.209147 16946 hybrid_clock.cc:648] HybridClock initialized: now 1790564326209101 us; error 34 us; skew 500 ppm
I20260928 02:58:46.209434 16946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:46 dist-test-slave-k5rg krb5kdc[16573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564326, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:46.212328 16946 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:58:46.213729 16946 webserver.cc:460] Webserver started at http://127.12.182.67:33787/ using document root <none> and password file <none>
I20260928 02:58:46.214367 16946 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:46.214478 16946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:46.214747 16946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:46.217506 16946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "bcbbef7adc1d4f6a83dda1963cb3ab96"
format_stamp: "Formatted at 2026-09-28 02:58:46 on dist-test-slave-k5rg"
server_key: "59b9d5bb372849e24b2d1e4cc0ebe7f9"
server_key_iv: "046f39b650b7d5bc741f1082dbcc3cdd"
server_key_version: "encryptionkey@0"
I20260928 02:58:46.218405 16946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "bcbbef7adc1d4f6a83dda1963cb3ab96"
format_stamp: "Formatted at 2026-09-28 02:58:46 on dist-test-slave-k5rg"
server_key: "59b9d5bb372849e24b2d1e4cc0ebe7f9"
server_key_iv: "046f39b650b7d5bc741f1082dbcc3cdd"
server_key_version: "encryptionkey@0"
I20260928 02:58:46.223305 16946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:58:46.226528 16961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:46.227747 16946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:46.227926 16946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "bcbbef7adc1d4f6a83dda1963cb3ab96"
format_stamp: "Formatted at 2026-09-28 02:58:46 on dist-test-slave-k5rg"
server_key: "59b9d5bb372849e24b2d1e4cc0ebe7f9"
server_key_iv: "046f39b650b7d5bc741f1082dbcc3cdd"
server_key_version: "encryptionkey@0"
I20260928 02:58:46.228101 16946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:46.246994 16946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:46.251187 16946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:46.251456 16946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:46.252449 16946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:46.253855 16946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:46.253935 16946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:46.254057 16946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:46.254112 16946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:46.283638 16946 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:42517
I20260928 02:58:46.283658 17073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:42517 every 8 connection(s)
I20260928 02:58:46.284888 16946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:58:46.287662 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 16946
I20260928 02:58:46.287835 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:58:46.288246 13017 external_mini_cluster.cc:1428] Setting key 7393ff911d0263c861073466eac1cdd3
Sep 28 02:58:46 dist-test-slave-k5rg krb5kdc[16573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564326, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:46.303498 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:46.287283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:39493 (local address 127.12.182.126:41225)
0928 02:58:46.287490 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:46.287497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:46.288779 (+ 1282us) server_negotiation.cc:407] Connection header received
0928 02:58:46.290622 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:46.290627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:46.290708 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:46.290804 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:46.292272 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.292969 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.294467 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.294868 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.294958 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:46.297970 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:46.297975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:46.297978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:46.298008 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:46.300400 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.301050 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.301054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.301056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.301168 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.301664 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.301669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.301672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.302054 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:46.302194 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:46.303004 (+ 810us) server_negotiation.cc:299] Negotiation successful
0928 02:58:46.303250 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:58:46.304462 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:46.287792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41225 (local address 127.12.182.67:39493)
0928 02:58:46.288608 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:46.288652 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:46.290373 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:46.291052 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:46.291065 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:46.291584 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:46.292142 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:46.292159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.293090 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.293093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:46.294342 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:46.294354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.295029 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.295033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:46.295215 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:46.295782 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:46.295802 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:46.297792 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:46.300538 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:46.300544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:46.300556 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:46.300941 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:46.301254 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:46.301259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:46.301262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:46.301505 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:46.302181 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:46.302189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:46.302585 (+ 396us) client_negotiation.cc:769] Sending connection context
0928 02:58:46.302883 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 02:58:46.303370 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":320,"threads_started":1}
I20260928 02:58:46.305785 17074 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41225
I20260928 02:58:46.306123 17074 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:46.306842 17074 heartbeater.cc:511] Master 127.12.182.126:41225 requested a full tablet report, sending...
I20260928 02:58:46.308228 16618 ts_manager.cc:194] Registered new tserver with Master: bcbbef7adc1d4f6a83dda1963cb3ab96 (127.12.182.67:42517)
I20260928 02:58:46.308987 16618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:39493
I20260928 02:58:46.315584 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:58:46 dist-test-slave-k5rg krb5kdc[16573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:58:46 dist-test-slave-k5rg krb5kdc[16573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:58:46.358646 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:46.344704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40862 (local address 127.12.182.65:35937)
0928 02:58:46.345297 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:46.345305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:46.345331 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:58:46.345444 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:46.345451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:46.345694 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:46.345824 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:46.347142 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.348124 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.349290 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.349920 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.350018 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:46.353584 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:46.353594 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:46.353632 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:46.353674 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:46.356173 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.356806 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.356814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.356817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.356935 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.357262 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.357265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.357267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.357639 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:46.357871 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:46.358176 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 02:58:46.358370 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":351,"threads_started":1}
Sep 28 02:58:46 dist-test-slave-k5rg krb5kdc[16573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:46.376457 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:46.362749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.12.182.126:41225)
0928 02:58:46.362951 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:46.362956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:46.363144 (+ 188us) server_negotiation.cc:407] Connection header received
0928 02:58:46.363286 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:46.363290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:46.363356 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:46.363435 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:46.364935 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.365737 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.367430 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.367836 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.367922 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:46.372139 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:46.372144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:46.372147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:46.372181 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:46.374485 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.375021 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.375024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.375026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.375141 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.375408 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.375411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.375413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.375728 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:46.375904 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:46.376096 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:58:46.376245 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 02:58:46.377004 16618 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49912
I20260928 02:58:46.377905 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16679
I20260928 02:58:46.389474 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:46.378705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.12.182.65:35937)
0928 02:58:46.378875 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:46.378881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:46.379135 (+ 254us) server_negotiation.cc:407] Connection header received
0928 02:58:46.379279 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:46.379283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:46.379352 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:46.379430 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:46.380516 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.381174 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.382358 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.382748 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.382852 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:46.384893 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:46.384897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:46.384901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:46.384929 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:46.387289 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.387838 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.387842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.387844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.387950 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.388275 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.388280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.388283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.388511 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:46.388727 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:46.388993 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:58:46.389190 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:58:46.390015 16802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:46.392534 16807 heartbeater.cc:503] Master 127.12.182.126:41225 was elected leader, sending a full tablet report...
I20260928 02:58:46.469511 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 16679
W20260928 02:58:46.482456 16604 connection.cc:535] server connection from 127.12.182.65:39665 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:58:46.482676 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16813
Sep 28 02:58:46 dist-test-slave-k5rg krb5kdc[16573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 02:58:46.500273 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:46.483489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.12.182.66:40497)
0928 02:58:46.484071 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:46.484078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:46.484102 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:58:46.484159 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:46.484165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:46.484376 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:46.484554 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:46.485707 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.486695 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.487907 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.488446 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.488571 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:46.495201 (+ 6630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:46.495212 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:46.495226 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:46.495258 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:46.497689 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.498291 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.498298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.498300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.498425 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.498772 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.498776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.498778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.499111 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:46.499398 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:46.499699 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 02:58:46.499947 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":279,"threads_started":1}
I20260928 02:58:46.501477 16936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:46.582504 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 16813
I20260928 02:58:46.594961 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16946
Sep 28 02:58:46 dist-test-slave-k5rg krb5kdc[16573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 02:58:46.610368 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:46.595915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.12.182.67:42517)
0928 02:58:46.596477 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:46.596483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:46.596529 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:58:46.596589 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:46.596595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:46.596794 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:46.596912 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:46.598212 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.599400 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.600635 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:46.601220 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:46.601311 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:46.605304 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:46.605315 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:46.605329 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:46.605360 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:46.607947 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.608531 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.608536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.608539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.608644 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:46.608983 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:46.608986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:46.608987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:46.609293 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:46.609506 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:46.609887 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 02:58:46.610087 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":275,"threads_started":1}
I20260928 02:58:46.611603 17069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:46.692243 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 16946
I20260928 02:58:46.704380 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 16586
I20260928 02:58:46.705428 16648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:46.807402 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 16586
2026-09-28T02:58:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3558 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:46 dist-test-slave-k5rg krb5kdc[17093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:46 dist-test-slave-k5rg krb5kdc[17093](info): set up 2 sockets
Sep 28 02:58:46 dist-test-slave-k5rg krb5kdc[17093](info): commencing operation
krb5kdc: starting...
W20260928 02:58:47.793937 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:58:47 dist-test-slave-k5rg krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:58:47.970861 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:46809
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:46809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:48.271325 17106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:48.271600 17106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:48.271700 17106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:48.278654 17106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:48.278748 17106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:48.278791 17106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:48.278846 17106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:48.278891 17106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:48.288412 17106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:46809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:46809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:48.289625 17106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:48.290839 17106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:48.301435 17111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:48.301463 17114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:48.301488 17112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:48.302240 17106 server_base.cc:1034] running on GCE node
I20260928 02:58:48.302963 17106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:48.304276 17106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:48.305476 17106 hybrid_clock.cc:648] HybridClock initialized: now 1790564328305427 us; error 47 us; skew 500 ppm
I20260928 02:58:48.305841 17106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:48 dist-test-slave-k5rg krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564328, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:48.308640 17106 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:58:48.309757 17106 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:48.309861 17106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:48.310169 17106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:48.313314 17106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "59831ddbf44a4ff393ba5eda0d540d5a"
format_stamp: "Formatted at 2026-09-28 02:58:48 on dist-test-slave-k5rg"
server_key: "7ffc862b36bbf0b411970e874059ea49"
server_key_iv: "6fbdd568dfb65d9c180f177e309b1520"
server_key_version: "encryptionkey@0"
I20260928 02:58:48.314129 17106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "59831ddbf44a4ff393ba5eda0d540d5a"
format_stamp: "Formatted at 2026-09-28 02:58:48 on dist-test-slave-k5rg"
server_key: "7ffc862b36bbf0b411970e874059ea49"
server_key_iv: "6fbdd568dfb65d9c180f177e309b1520"
server_key_version: "encryptionkey@0"
I20260928 02:58:48.318876 17106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 02:58:48.322279 17120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:48.323467 17106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:48.323623 17106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "59831ddbf44a4ff393ba5eda0d540d5a"
format_stamp: "Formatted at 2026-09-28 02:58:48 on dist-test-slave-k5rg"
server_key: "7ffc862b36bbf0b411970e874059ea49"
server_key_iv: "6fbdd568dfb65d9c180f177e309b1520"
server_key_version: "encryptionkey@0"
I20260928 02:58:48.323803 17106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:48.354283 17106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:48.358174 17106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:48.358418 17106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:48.374648 17106 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:46809
I20260928 02:58:48.374668 17171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:46809 every 8 connection(s)
I20260928 02:58:48.375447 17106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:48.376536 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 17106
I20260928 02:58:48.376760 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:48.377508 13017 external_mini_cluster.cc:1428] Setting key 55d6ac011c91da9e3bbd24ad6a73c063
I20260928 02:58:48.379763 17172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:58:48 dist-test-slave-k5rg krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:48.391611 17172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a: Bootstrap starting.
I20260928 02:58:48.395289 17172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:48.396445 17172 log.cc:824] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:48.399314 17172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a: No bootstrap required, opened a new log
I20260928 02:58:48.400104 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:48.379578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.12.182.126:46809)
0928 02:58:48.380366 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:48.380377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:48.380413 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:58:48.382109 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:48.382136 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:48.382632 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:48.383045 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:48.384322 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:48.386334 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:48.387535 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:48.388527 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:48.388712 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:48.392384 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:48.392394 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:48.392411 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:48.392451 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:48.395988 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:48.396665 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:48.396672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:48.396677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:48.396845 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:48.397293 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:48.397297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:48.397300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:48.397959 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:48.398157 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:48.398646 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 02:58:48.399048 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":597,"thread_start_us":358,"threads_started":1}
I20260928 02:58:48.406311 17172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59831ddbf44a4ff393ba5eda0d540d5a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46809 } }
I20260928 02:58:48.406646 17172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:48.406728 17172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59831ddbf44a4ff393ba5eda0d540d5a, State: Initialized, Role: FOLLOWER
I20260928 02:58:48.407202 17172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59831ddbf44a4ff393ba5eda0d540d5a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46809 } }
I20260928 02:58:48.407408 17172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:48.407526 17172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:48.407678 17172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:48.409739 17172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59831ddbf44a4ff393ba5eda0d540d5a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46809 } }
I20260928 02:58:48.410207 17172 leader_election.cc:302] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59831ddbf44a4ff393ba5eda0d540d5a; no voters:
I20260928 02:58:48.410764 17172 leader_election.cc:288] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:48.410873 17177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:48.411495 17177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [term 1 LEADER]: Becoming Leader. State: Replica: 59831ddbf44a4ff393ba5eda0d540d5a, State: Running, Role: LEADER
I20260928 02:58:48.411988 17177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59831ddbf44a4ff393ba5eda0d540d5a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46809 } }
I20260928 02:58:48.412672 17172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:48.414635 17178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59831ddbf44a4ff393ba5eda0d540d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59831ddbf44a4ff393ba5eda0d540d5a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46809 } } }
I20260928 02:58:48.415088 17178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:48.415354 17179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59831ddbf44a4ff393ba5eda0d540d5a. Latest consensus state: current_term: 1 leader_uuid: "59831ddbf44a4ff393ba5eda0d540d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59831ddbf44a4ff393ba5eda0d540d5a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46809 } } }
I20260928 02:58:48.415694 17179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:48.418672 17185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:48.423748 17185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:48.431972 17185 catalog_manager.cc:1347] Generated new cluster ID: d627bc9f938448a7a0d6c14ac9649eb9
I20260928 02:58:48.432094 17185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:48.445794 17185 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:48.447585 17185 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:48.463338 17185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a: Generated new TSK 0
I20260928 02:58:48.464442 17185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:58:48.506870 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46809
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:48.810477 17198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:48.810722 17198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:48.810860 17198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:48.818017 17198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:48.818111 17198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:48.818262 17198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:58:48.828204 17198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:48.829448 17198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:48.830691 17198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:48.842875 17206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:48.842875 17203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:48.842922 17204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:48.843736 17198 server_base.cc:1034] running on GCE node
I20260928 02:58:48.844256 17198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:48.845093 17198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:48.846274 17198 hybrid_clock.cc:648] HybridClock initialized: now 1790564328846222 us; error 44 us; skew 500 ppm
I20260928 02:58:48.846565 17198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:48 dist-test-slave-k5rg krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564328, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:48.849442 17198 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:58:48.850476 17198 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:48.850561 17198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:48.850805 17198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:48.853539 17198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "5227ff9163c646fca5d05eb663c7478c"
format_stamp: "Formatted at 2026-09-28 02:58:48 on dist-test-slave-k5rg"
server_key: "cd7cec3390e99972f3de461ae74f042a"
server_key_iv: "ea497c5ac85f3eff883097f23e75abf3"
server_key_version: "encryptionkey@0"
I20260928 02:58:48.854303 17198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "5227ff9163c646fca5d05eb663c7478c"
format_stamp: "Formatted at 2026-09-28 02:58:48 on dist-test-slave-k5rg"
server_key: "cd7cec3390e99972f3de461ae74f042a"
server_key_iv: "ea497c5ac85f3eff883097f23e75abf3"
server_key_version: "encryptionkey@0"
I20260928 02:58:48.859040 17198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:58:48.862310 17212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:48.863536 17198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:58:48.863678 17198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "5227ff9163c646fca5d05eb663c7478c"
format_stamp: "Formatted at 2026-09-28 02:58:48 on dist-test-slave-k5rg"
server_key: "cd7cec3390e99972f3de461ae74f042a"
server_key_iv: "ea497c5ac85f3eff883097f23e75abf3"
server_key_version: "encryptionkey@0"
I20260928 02:58:48.863878 17198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:48.884761 17198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:48.888779 17198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:48.889024 17198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:48.889946 17198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:48.891249 17198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:48.891345 17198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:48.891456 17198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:48.891522 17198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:48.919281 17198 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:35041
I20260928 02:58:48.919318 17324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:35041 every 8 connection(s)
I20260928 02:58:48.920039 17198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:58:48.922688 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 17198
I20260928 02:58:48.922879 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:58:48.923465 13017 external_mini_cluster.cc:1428] Setting key e756c619bac3b358d9f46c30cd652e00
Sep 28 02:58:48 dist-test-slave-k5rg krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564328, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
I20260928 02:58:48.943028 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:48.922199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:59363 (local address 127.12.182.126:46809)
0928 02:58:48.922679 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:48.922686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:48.923656 (+ 970us) server_negotiation.cc:407] Connection header received
0928 02:58:48.925659 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:48.925665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:48.925738 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:48.925854 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:48.927322 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:48.928266 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:48.929785 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:48.930348 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:48.930443 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:48.934738 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:48.934746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:48.934752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:48.934792 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:48.939874 (+ 5082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:48.940779 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:48.940785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:48.940789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:48.940911 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:48.941303 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:48.941308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:48.941312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:48.941591 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:48.941753 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:48.942452 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 02:58:48.942678 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
I20260928 02:58:48.944034 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:48.922740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46809 (local address 127.12.182.65:59363)
0928 02:58:48.923489 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:48.923540 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:48.925394 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:48.926056 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:48.926063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:48.926560 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:48.927164 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:48.927182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:48.928398 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:48.928402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:48.929652 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:48.929665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:48.930565 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:48.930571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:48.930831 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:48.931536 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:48.931567 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:48.934435 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:48.940049 (+ 5614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:48.940059 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:48.940078 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:48.940668 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:48.941016 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:48.941021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:48.941025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:48.941199 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:48.941729 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:48.941736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:48.942068 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 02:58:48.942333 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 02:58:48.942871 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":311,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
I20260928 02:58:48.945572 17325 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46809
I20260928 02:58:48.945879 17325 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:48.946483 17325 heartbeater.cc:511] Master 127.12.182.126:46809 requested a full tablet report, sending...
I20260928 02:58:48.948424 17137 ts_manager.cc:194] Registered new tserver with Master: 5227ff9163c646fca5d05eb663c7478c (127.12.182.65:35041)
I20260928 02:58:48.949877 17137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:59363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:58:48.965029 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46809
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:49.267686 17331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:49.267980 17331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:49.268087 17331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:49.275014 17331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:49.275108 17331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:49.275220 17331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:58:49.284515 17331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:49.285733 17331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:49.286926 17331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:49.298861 17336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:49.298839 17337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:49.298890 17339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:49.299779 17331 server_base.cc:1034] running on GCE node
I20260928 02:58:49.300333 17331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:49.301198 17331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:49.302397 17331 hybrid_clock.cc:648] HybridClock initialized: now 1790564329302355 us; error 37 us; skew 500 ppm
I20260928 02:58:49.302690 17331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:49 dist-test-slave-k5rg krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564329, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:49.305536 17331 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:58:49.306569 17331 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:49.306666 17331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:49.306948 17331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:49.309725 17331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "7974e03ac3e54b57b21dde2020d45070"
format_stamp: "Formatted at 2026-09-28 02:58:49 on dist-test-slave-k5rg"
server_key: "05a520718d1d8252990ad0fd4f5bcdb0"
server_key_iv: "e689440987e2b04b40d478ed3fc9f704"
server_key_version: "encryptionkey@0"
I20260928 02:58:49.310491 17331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "7974e03ac3e54b57b21dde2020d45070"
format_stamp: "Formatted at 2026-09-28 02:58:49 on dist-test-slave-k5rg"
server_key: "05a520718d1d8252990ad0fd4f5bcdb0"
server_key_iv: "e689440987e2b04b40d478ed3fc9f704"
server_key_version: "encryptionkey@0"
I20260928 02:58:49.315310 17331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:58:49.318562 17345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:49.319811 17331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:58:49.319952 17331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "7974e03ac3e54b57b21dde2020d45070"
format_stamp: "Formatted at 2026-09-28 02:58:49 on dist-test-slave-k5rg"
server_key: "05a520718d1d8252990ad0fd4f5bcdb0"
server_key_iv: "e689440987e2b04b40d478ed3fc9f704"
server_key_version: "encryptionkey@0"
I20260928 02:58:49.320142 17331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:49.350153 17331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:49.353984 17331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:49.354238 17331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:49.355134 17331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:49.356485 17331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:49.356578 17331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:49.356676 17331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:49.356746 17331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:49.386389 17331 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:41687
I20260928 02:58:49.386482 17457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:41687 every 8 connection(s)
I20260928 02:58:49.387156 17331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:58:49.391201 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 17331
I20260928 02:58:49.391392 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:58:49.391907 13017 external_mini_cluster.cc:1428] Setting key 2f8f0a5ba737a878b320fad76571e79a
Sep 28 02:58:49 dist-test-slave-k5rg krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564329, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:49.407368 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.389538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:53325 (local address 127.12.182.126:46809)
0928 02:58:49.389721 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:49.389726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:49.390986 (+ 1260us) server_negotiation.cc:407] Connection header received
0928 02:58:49.393257 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:49.393262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:49.393326 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:49.393406 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:49.395246 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.395941 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.397921 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.398309 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.398397 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.402057 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:49.402062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:49.402066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:49.402096 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:49.404466 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:49.405254 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:49.405258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:49.405260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:49.405374 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:49.405810 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:49.405813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:49.405815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:49.406104 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:49.406212 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:49.406983 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 02:58:49.407149 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:58:49.408774 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.390111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46809 (local address 127.12.182.66:53325)
0928 02:58:49.390817 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:49.390859 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:49.392994 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:49.393658 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:49.393670 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:49.394343 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:49.395071 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:49.395095 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.396080 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.396085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:49.397787 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:49.397802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.398505 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.398511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:49.398750 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.399376 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:49.399407 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:49.401817 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:49.404589 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:49.404597 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:49.404613 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:49.405152 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:49.405462 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:49.405466 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:49.405469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:49.405718 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:49.406222 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:49.406230 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:49.406629 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 02:58:49.406954 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 02:58:49.407458 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":303,"threads_started":1}
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
I20260928 02:58:49.410161 17458 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46809
I20260928 02:58:49.410431 17458 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:49.411016 17458 heartbeater.cc:511] Master 127.12.182.126:46809 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
I20260928 02:58:49.412293 17137 ts_manager.cc:194] Registered new tserver with Master: 7974e03ac3e54b57b21dde2020d45070 (127.12.182.66:41687)
I20260928 02:58:49.413331 17137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:53325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:58:49.432842 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46809
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:49.732492 17463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:49.732751 17463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:49.732896 17463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:49.739959 17463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:49.740074 17463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:49.740231 17463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:58:49.749651 17463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:49.750887 17463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:49.752118 17463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:49.763909 17468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:49.763952 17469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:49.763973 17471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:49.764760 17463 server_base.cc:1034] running on GCE node
I20260928 02:58:49.765244 17463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:49.766113 17463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:49.767297 17463 hybrid_clock.cc:648] HybridClock initialized: now 1790564329767246 us; error 46 us; skew 500 ppm
I20260928 02:58:49.767570 17463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:49 dist-test-slave-k5rg krb5kdc[17093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564329, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:49.770439 17463 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:58:49.771420 17463 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:49.771503 17463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:49.771739 17463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:49.774474 17463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "5b793a0c18924c109b904e25e8cd0ac4"
format_stamp: "Formatted at 2026-09-28 02:58:49 on dist-test-slave-k5rg"
server_key: "1a6f8cbe1ca58a05705a73039ee6a89a"
server_key_iv: "dcf86746c62bd7d9537d57a126ea64b9"
server_key_version: "encryptionkey@0"
I20260928 02:58:49.775189 17463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "5b793a0c18924c109b904e25e8cd0ac4"
format_stamp: "Formatted at 2026-09-28 02:58:49 on dist-test-slave-k5rg"
server_key: "1a6f8cbe1ca58a05705a73039ee6a89a"
server_key_iv: "dcf86746c62bd7d9537d57a126ea64b9"
server_key_version: "encryptionkey@0"
I20260928 02:58:49.779708 17463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 02:58:49.782732 17477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:49.783874 17463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:58:49.784026 17463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "5b793a0c18924c109b904e25e8cd0ac4"
format_stamp: "Formatted at 2026-09-28 02:58:49 on dist-test-slave-k5rg"
server_key: "1a6f8cbe1ca58a05705a73039ee6a89a"
server_key_iv: "dcf86746c62bd7d9537d57a126ea64b9"
server_key_version: "encryptionkey@0"
I20260928 02:58:49.784230 17463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:49.798554 17463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:49.802055 17463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:49.802286 17463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:49.803103 17463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:49.804342 17463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:49.804426 17463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:49.804525 17463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:49.804589 17463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:49.832918 17463 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:33065
I20260928 02:58:49.832952 17589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:33065 every 8 connection(s)
I20260928 02:58:49.833847 17463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:58:49.837357 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 17463
I20260928 02:58:49.837569 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:58:49.838337 13017 external_mini_cluster.cc:1428] Setting key 3045a694368fa02f5a705929b4cc82b0
Sep 28 02:58:49 dist-test-slave-k5rg krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564329, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:49.858922 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.837555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46809 (local address 127.12.182.67:56169)
0928 02:58:49.838398 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:49.838441 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:49.840255 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:49.841133 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:49.841145 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:49.841906 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:49.842637 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:49.842660 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.844259 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.844265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:49.846216 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:49.846231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.847364 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.847372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:49.847600 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.848316 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:49.848352 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:49.851053 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:49.854804 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:49.854811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:49.854827 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:49.855329 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:49.855674 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:49.855679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:49.855683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:49.855879 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:49.856404 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:49.856411 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:49.856799 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 02:58:49.857075 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 02:58:49.857585 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":355,"threads_started":1}
I20260928 02:58:49.859902 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.837322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:56169 (local address 127.12.182.126:46809)
0928 02:58:49.837579 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:49.837586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:49.838543 (+ 957us) server_negotiation.cc:407] Connection header received
0928 02:58:49.840550 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:49.840555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:49.840639 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:49.840730 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:49.842804 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.844099 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.846515 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.847144 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.847271 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.851321 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:49.851327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:49.851333 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:49.851374 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:49.854647 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:49.855444 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:49.855449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:49.855453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:49.855574 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:49.855990 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:49.855995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:49.855999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:49.856274 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:49.856403 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:49.859381 (+ 2978us) server_negotiation.cc:299] Negotiation successful
0928 02:58:49.859592 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 02:58:49.864805 17590 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46809
I20260928 02:58:49.865114 17590 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:49.865873 17590 heartbeater.cc:511] Master 127.12.182.126:46809 requested a full tablet report, sending...
I20260928 02:58:49.867295 17137 ts_manager.cc:194] Registered new tserver with Master: 5b793a0c18924c109b904e25e8cd0ac4 (127.12.182.67:33065)
I20260928 02:58:49.868366 17137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:56169
I20260928 02:58:49.878639 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:58:49.894084 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.883024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.12.182.126:46809)
0928 02:58:49.883172 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:49.883176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:49.883615 (+ 439us) server_negotiation.cc:407] Connection header received
0928 02:58:49.883740 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:49.883744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:49.883807 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:49.883897 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:49.885123 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.885854 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.887037 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.887477 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.887564 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.889663 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:49.889668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:49.889672 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:49.889704 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:49.892053 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:49.892630 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:49.892633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:49.892636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:49.892749 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:49.893017 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:49.893020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:49.893022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:49.893290 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:49.893467 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:49.893719 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:58:49.893863 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:58:49.897696 17137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:58:49.904102 17137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:49.926152 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.915020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:33065 (local address 127.0.0.1:37682)
0928 02:58:49.916547 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:49.916602 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:49.916853 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:49.917910 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:49.917918 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:49.917949 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:49.918348 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:49.918360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.920691 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.920696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:49.923110 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:49.923125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.925234 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.925240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:49.925481 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.925535 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:58:49.925666 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 02:58:49.925830 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":23,"threads_started":1}
I20260928 02:58:49.926987 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.915251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.12.182.67:33065)
0928 02:58:49.915995 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:49.916003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:49.916704 (+ 701us) server_negotiation.cc:407] Connection header received
0928 02:58:49.917083 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:49.917092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:49.917349 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:49.917516 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:49.919010 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.920546 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.923288 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.924779 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.924890 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.925106 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:49.926517 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0928 02:58:49.926679 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":723,"threads_started":1}
I20260928 02:58:49.928256 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.915377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:41687 (local address 127.0.0.1:41492)
0928 02:58:49.919895 (+ 4518us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:49.919916 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:49.920095 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:49.920870 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:49.920875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:49.920913 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:49.921348 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:49.921361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.923335 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.923340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:49.925518 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:49.925531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.927500 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.927506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:49.927674 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.927699 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:58:49.927766 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:58:49.927876 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4415,"mutex_wait_us":38,"thread_start_us":4790,"threads_started":1}
I20260928 02:58:49.928296 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.915770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.12.182.66:41687)
0928 02:58:49.918893 (+ 3123us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:49.918902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:49.920002 (+ 1100us) server_negotiation.cc:407] Connection header received
0928 02:58:49.920246 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:49.920253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:49.920461 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:49.920612 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:49.921560 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.923142 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.925709 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.927028 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.927121 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.927287 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:49.927827 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 02:58:49.927998 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2970,"thread_start_us":2867,"threads_started":1}
I20260928 02:58:49.932129 17525 tablet_service.cc:1467] Processing CreateTablet for tablet 7c173f80f44b42f98a5cb1c573754360 (DEFAULT_TABLE table=test-table [id=0d900ee7a634488ebdc90260aeb21a4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:49.932776 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.915998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.12.182.65:35041)
0928 02:58:49.919521 (+ 3523us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:49.919529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:49.926438 (+ 6909us) server_negotiation.cc:407] Connection header received
0928 02:58:49.926678 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:49.926685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:49.926891 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:49.927054 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:49.927650 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.929202 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.930816 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.931793 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.931916 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.932123 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:49.932317 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 02:58:49.932475 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3361,"thread_start_us":3973,"threads_started":1}
I20260928 02:58:49.932127 17393 tablet_service.cc:1467] Processing CreateTablet for tablet 7c173f80f44b42f98a5cb1c573754360 (DEFAULT_TABLE table=test-table [id=0d900ee7a634488ebdc90260aeb21a4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:49.933341 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:49.915526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35041 (local address 127.0.0.1:47094)
0928 02:58:49.926348 (+ 10822us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:49.926363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:49.926521 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:49.927296 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:49.927301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:49.927324 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:49.927499 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:49.927508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.929364 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.929369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:49.930664 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:49.930675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:49.931943 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:49.931949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:49.932123 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:49.932154 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:58:49.932942 (+ 788us) client_negotiation.cc:240] Negotiation successful
0928 02:58:49.933109 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10725,"thread_start_us":635,"threads_started":1}
I20260928 02:58:49.933560 17525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c173f80f44b42f98a5cb1c573754360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:49.933797 17393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c173f80f44b42f98a5cb1c573754360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:49.936151 17260 tablet_service.cc:1467] Processing CreateTablet for tablet 7c173f80f44b42f98a5cb1c573754360 (DEFAULT_TABLE table=test-table [id=0d900ee7a634488ebdc90260aeb21a4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:49.937165 17260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c173f80f44b42f98a5cb1c573754360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:49.947396 17608 tablet_bootstrap.cc:492] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: Bootstrap starting.
I20260928 02:58:49.950300 17609 tablet_bootstrap.cc:492] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: Bootstrap starting.
I20260928 02:58:49.950836 17608 tablet_bootstrap.cc:654] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:49.951668 17610 tablet_bootstrap.cc:492] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: Bootstrap starting.
I20260928 02:58:49.952207 17608 log.cc:824] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:49.952523 17325 heartbeater.cc:503] Master 127.12.182.126:46809 was elected leader, sending a full tablet report...
I20260928 02:58:49.953595 17609 tablet_bootstrap.cc:654] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:49.954825 17609 log.cc:824] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:49.955438 17610 tablet_bootstrap.cc:654] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:49.956570 17610 log.cc:824] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:49.959372 17610 tablet_bootstrap.cc:492] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: No bootstrap required, opened a new log
I20260928 02:58:49.959496 17609 tablet_bootstrap.cc:492] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: No bootstrap required, opened a new log
I20260928 02:58:49.959486 17608 tablet_bootstrap.cc:492] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: No bootstrap required, opened a new log
I20260928 02:58:49.959613 17610 ts_tablet_manager.cc:1397] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 02:58:49.959738 17608 ts_tablet_manager.cc:1397] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260928 02:58:49.959733 17609 ts_tablet_manager.cc:1397] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:58:49.966495 17610 raft_consensus.cc:348] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 } } peers { permanent_uuid: "7974e03ac3e54b57b21dde2020d45070" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41687 } } peers { permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 } }
I20260928 02:58:49.966814 17610 raft_consensus.cc:374] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:49.966900 17610 raft_consensus.cc:729] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b793a0c18924c109b904e25e8cd0ac4, State: Initialized, Role: FOLLOWER
I20260928 02:58:49.967466 17610 consensus_queue.cc:260] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 } } peers { permanent_uuid: "7974e03ac3e54b57b21dde2020d45070" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41687 } } peers { permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 } }
I20260928 02:58:49.968642 17590 heartbeater.cc:503] Master 127.12.182.126:46809 was elected leader, sending a full tablet report...
I20260928 02:58:49.969152 17610 ts_tablet_manager.cc:1428] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20260928 02:58:49.969630 17609 raft_consensus.cc:348] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 } } peers { permanent_uuid: "7974e03ac3e54b57b21dde2020d45070" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41687 } } peers { permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 } }
I20260928 02:58:49.969622 17608 raft_consensus.cc:348] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 } } peers { permanent_uuid: "7974e03ac3e54b57b21dde2020d45070" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41687 } } peers { permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 } }
I20260928 02:58:49.970021 17609 raft_consensus.cc:374] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:49.970021 17608 raft_consensus.cc:374] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:49.970120 17608 raft_consensus.cc:729] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227ff9163c646fca5d05eb663c7478c, State: Initialized, Role: FOLLOWER
I20260928 02:58:49.970120 17609 raft_consensus.cc:729] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7974e03ac3e54b57b21dde2020d45070, State: Initialized, Role: FOLLOWER
I20260928 02:58:49.970671 17608 consensus_queue.cc:260] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 } } peers { permanent_uuid: "7974e03ac3e54b57b21dde2020d45070" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41687 } } peers { permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 } }
I20260928 02:58:49.970672 17609 consensus_queue.cc:260] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 } } peers { permanent_uuid: "7974e03ac3e54b57b21dde2020d45070" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41687 } } peers { permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 } }
I20260928 02:58:49.972194 17458 heartbeater.cc:503] Master 127.12.182.126:46809 was elected leader, sending a full tablet report...
I20260928 02:58:49.972338 17609 ts_tablet_manager.cc:1428] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260928 02:58:49.974766 17608 ts_tablet_manager.cc:1428] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: Time spent starting tablet: real 0.015s user 0.001s sys 0.015s
W20260928 02:58:50.085637 17591 tablet.cc:2369] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:58:50.138978 17459 tablet.cc:2369] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:58:50.172183 17326 tablet.cc:2369] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:58:50.381739 17615 raft_consensus.cc:482] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:50.382071 17615 raft_consensus.cc:504] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 } } peers { permanent_uuid: "7974e03ac3e54b57b21dde2020d45070" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41687 } } peers { permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 } }
I20260928 02:58:50.383723 17615 leader_election.cc:288] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b793a0c18924c109b904e25e8cd0ac4 (127.12.182.67:33065), 5227ff9163c646fca5d05eb663c7478c (127.12.182.65:35041)
I20260928 02:58:50.392364 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:50.384197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:39407 (local address 127.12.182.65:35041)
0928 02:58:50.384418 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:50.384424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:50.385753 (+ 1329us) server_negotiation.cc:407] Connection header received
0928 02:58:50.386010 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:50.386015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:50.386090 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:50.386151 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:50.387109 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.388150 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.390796 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.391518 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.391663 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:50.391735 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:50.392014 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:58:50.392135 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:58:50.392748 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:50.384263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35041 (local address 127.12.182.66:39407)
0928 02:58:50.385670 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:50.385686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:50.385842 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:50.386324 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:50.386328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:50.386362 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:50.386872 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:50.386884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.388331 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.388335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:50.390608 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:50.390624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.391721 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.391727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:50.391858 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:50.391883 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:58:50.391963 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:58:50.392085 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"mutex_wait_us":30,"spinlock_wait_cycles":23808,"thread_start_us":744,"threads_started":1}
I20260928 02:58:50.393064 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:50.384630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:53737 (local address 127.12.182.67:33065)
0928 02:58:50.384842 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:50.384849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:50.385442 (+ 593us) server_negotiation.cc:407] Connection header received
0928 02:58:50.385786 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:50.385792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:50.385868 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:50.385946 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:58:50.386869 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.387974 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.390765 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.391967 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.392115 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:50.392183 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:50.392705 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 02:58:50.392837 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:58:50.393179 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:50.384133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:33065 (local address 127.12.182.66:53737)
0928 02:58:50.385320 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:50.385340 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:50.385570 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:50.386212 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:50.386216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:50.386250 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:58:50.386687 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:50.386699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.388211 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.388216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:50.390609 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:50.390624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.392333 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.392339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:50.392533 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:50.392564 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:58:50.392647 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:58:50.392776 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":68,"thread_start_us":1197,"threads_started":1}
I20260928 02:58:50.394076 17545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c173f80f44b42f98a5cb1c573754360" candidate_uuid: "7974e03ac3e54b57b21dde2020d45070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b793a0c18924c109b904e25e8cd0ac4" is_pre_election: true
I20260928 02:58:50.394084 17280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c173f80f44b42f98a5cb1c573754360" candidate_uuid: "7974e03ac3e54b57b21dde2020d45070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227ff9163c646fca5d05eb663c7478c" is_pre_election: true
I20260928 02:58:50.394558 17545 raft_consensus.cc:2454] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7974e03ac3e54b57b21dde2020d45070 in term 0.
I20260928 02:58:50.394558 17280 raft_consensus.cc:2454] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7974e03ac3e54b57b21dde2020d45070 in term 0.
I20260928 02:58:50.395155 17346 leader_election.cc:302] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5227ff9163c646fca5d05eb663c7478c, 5b793a0c18924c109b904e25e8cd0ac4, 7974e03ac3e54b57b21dde2020d45070; no voters:
I20260928 02:58:50.395540 17615 raft_consensus.cc:2781] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:50.395692 17615 raft_consensus.cc:482] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:50.395792 17615 raft_consensus.cc:3037] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:50.398921 17615 raft_consensus.cc:504] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 } } peers { permanent_uuid: "7974e03ac3e54b57b21dde2020d45070" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41687 } } peers { permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 } }
I20260928 02:58:50.399539 17615 leader_election.cc:288] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [CANDIDATE]: Term 1 election: Requested vote from peers 5b793a0c18924c109b904e25e8cd0ac4 (127.12.182.67:33065), 5227ff9163c646fca5d05eb663c7478c (127.12.182.65:35041)
I20260928 02:58:50.399971 17280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c173f80f44b42f98a5cb1c573754360" candidate_uuid: "7974e03ac3e54b57b21dde2020d45070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5227ff9163c646fca5d05eb663c7478c"
I20260928 02:58:50.399950 17545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c173f80f44b42f98a5cb1c573754360" candidate_uuid: "7974e03ac3e54b57b21dde2020d45070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b793a0c18924c109b904e25e8cd0ac4"
I20260928 02:58:50.400141 17280 raft_consensus.cc:3037] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:50.400138 17545 raft_consensus.cc:3037] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:50.402232 17280 raft_consensus.cc:2454] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7974e03ac3e54b57b21dde2020d45070 in term 1.
I20260928 02:58:50.402287 17545 raft_consensus.cc:2454] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7974e03ac3e54b57b21dde2020d45070 in term 1.
I20260928 02:58:50.402626 17347 leader_election.cc:302] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5227ff9163c646fca5d05eb663c7478c, 5b793a0c18924c109b904e25e8cd0ac4, 7974e03ac3e54b57b21dde2020d45070; no voters:
I20260928 02:58:50.402995 17615 raft_consensus.cc:2781] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:50.403352 17615 raft_consensus.cc:686] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 1 LEADER]: Becoming Leader. State: Replica: 7974e03ac3e54b57b21dde2020d45070, State: Running, Role: LEADER
I20260928 02:58:50.403869 17615 consensus_queue.cc:237] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 } } peers { permanent_uuid: "7974e03ac3e54b57b21dde2020d45070" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41687 } } peers { permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 } }
I20260928 02:58:50.407867 17137 catalog_manager.cc:5495] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7974e03ac3e54b57b21dde2020d45070 (127.12.182.66). New cstate: current_term: 1 leader_uuid: "7974e03ac3e54b57b21dde2020d45070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7974e03ac3e54b57b21dde2020d45070" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260928 02:58:50.487861 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:50.482427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.12.182.66:41687)
0928 02:58:50.483084 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:50.483090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:50.483108 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:58:50.483195 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:50.483199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:50.483277 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:50.483407 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:50.483807 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.484473 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.485817 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.486228 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.486334 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:50.486780 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:50.487290 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:50.487420 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:50.487527 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:58:50.487608 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":360,"threads_started":1}
I20260928 02:58:50.495169 17280 raft_consensus.cc:1261] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c [term 1 FOLLOWER]: Refusing update from remote peer 7974e03ac3e54b57b21dde2020d45070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:50.495169 17545 raft_consensus.cc:1261] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4 [term 1 FOLLOWER]: Refusing update from remote peer 7974e03ac3e54b57b21dde2020d45070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:50.496065 17619 consensus_queue.cc:1035] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b793a0c18924c109b904e25e8cd0ac4" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:50.496383 17615 consensus_queue.cc:1035] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227ff9163c646fca5d05eb663c7478c" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:50.505733 17624 mvcc.cc:205] Tried to move back new op lower bound from 7334151497699495936 to 7334151497337450496. Current Snapshot: MvccSnapshot[applied={T|T < 7334151497699495936}]
I20260928 02:58:50.509867 17628 mvcc.cc:205] Tried to move back new op lower bound from 7334151497699495936 to 7334151497337450496. Current Snapshot: MvccSnapshot[applied={T|T < 7334151497699495936}]
I20260928 02:58:50.518971 17135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:50.519235 17135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:50.522646 17135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59831ddbf44a4ff393ba5eda0d540d5a: Sending DeleteTablet for 3 replicas of tablet 7c173f80f44b42f98a5cb1c573754360
I20260928 02:58:50.523674 17525 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c173f80f44b42f98a5cb1c573754360 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:50 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:37682
I20260928 02:58:50.523818 17260 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c173f80f44b42f98a5cb1c573754360 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:50 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:47094
I20260928 02:58:50.523834 17393 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c173f80f44b42f98a5cb1c573754360 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:50 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:41492
I20260928 02:58:50.527935 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 17198
I20260928 02:58:50.534657 17636 tablet_replica.cc:330] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: stopping tablet replica
I20260928 02:58:50.535032 17636 raft_consensus.cc:2229] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:50.535465 17636 raft_consensus.cc:2258] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:50.537585 17635 tablet_replica.cc:330] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: stopping tablet replica
I20260928 02:58:50.537945 17635 raft_consensus.cc:2229] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:50.538331 17635 raft_consensus.cc:2258] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:50.539698 17636 ts_tablet_manager.cc:1905] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:50.544421 17636 ts_tablet_manager.cc:1918] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:58:50.544572 17636 log.cc:1196] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/7c173f80f44b42f98a5cb1c573754360
I20260928 02:58:50.545046 17636 ts_tablet_manager.cc:1939] T 7c173f80f44b42f98a5cb1c573754360 P 5227ff9163c646fca5d05eb663c7478c: Deleting consensus metadata
I20260928 02:58:50.546141 17635 ts_tablet_manager.cc:1905] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:50.546952 17122 catalog_manager.cc:4841] TS 5227ff9163c646fca5d05eb663c7478c (127.12.182.65:35041): tablet 7c173f80f44b42f98a5cb1c573754360 (table test-table [id=0d900ee7a634488ebdc90260aeb21a4b]) successfully deleted
I20260928 02:58:50.548444 17637 tablet_replica.cc:330] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: stopping tablet replica
I20260928 02:58:50.548888 17637 raft_consensus.cc:2229] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:58:50.549552 17637 raft_consensus.cc:2258] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:50.551452 17637 ts_tablet_manager.cc:1905] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:50.553100 17635 ts_tablet_manager.cc:1918] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:58:50.553298 17635 log.cc:1196] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/7c173f80f44b42f98a5cb1c573754360
I20260928 02:58:50.553833 17635 ts_tablet_manager.cc:1939] T 7c173f80f44b42f98a5cb1c573754360 P 5b793a0c18924c109b904e25e8cd0ac4: Deleting consensus metadata
I20260928 02:58:50.555603 17121 catalog_manager.cc:4841] TS 5b793a0c18924c109b904e25e8cd0ac4 (127.12.182.67:33065): tablet 7c173f80f44b42f98a5cb1c573754360 (table test-table [id=0d900ee7a634488ebdc90260aeb21a4b]) successfully deleted
I20260928 02:58:50.557775 17637 ts_tablet_manager.cc:1918] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 28 02:58:50 dist-test-slave-k5rg krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:58:50.557950 17637 log.cc:1196] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/7c173f80f44b42f98a5cb1c573754360
I20260928 02:58:50.558496 17637 ts_tablet_manager.cc:1939] T 7c173f80f44b42f98a5cb1c573754360 P 7974e03ac3e54b57b21dde2020d45070: Deleting consensus metadata
I20260928 02:58:50.560045 17123 catalog_manager.cc:4841] TS 7974e03ac3e54b57b21dde2020d45070 (127.12.182.66:41687): tablet 7c173f80f44b42f98a5cb1c573754360 (table test-table [id=0d900ee7a634488ebdc90260aeb21a4b]) successfully deleted
I20260928 02:58:50.565831 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:50.529426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.12.182.65:35041)
0928 02:58:50.529597 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:50.529622 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:50.547791 (+ 18169us) server_negotiation.cc:407] Connection header received
0928 02:58:50.547979 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:50.547985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:50.548068 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:50.548174 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:50.549959 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.551105 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.552964 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.553583 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.553718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:50.559022 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:50.559040 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:50.559059 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:50.559114 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:50.562910 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:50.563653 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:50.563664 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:50.563667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:50.563869 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:50.564234 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:50.564239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:50.564243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:50.564708 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:50.565032 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:50.565195 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:58:50.565414 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:58:50.566498 17320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:50.690323 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 17198
I20260928 02:58:50.704167 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 17331
Sep 28 02:58:50 dist-test-slave-k5rg krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 02:58:50.720649 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:50.704954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.12.182.66:41687)
0928 02:58:50.705126 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:50.705130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:50.705150 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:58:50.705360 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:50.705364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:50.705432 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:50.705509 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:50.707104 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.707876 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.709723 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.710178 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.710274 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:50.714889 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:50.714904 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:50.714946 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:50.714985 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:50.717933 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:50.718591 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:50.718598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:50.718600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:50.718775 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:50.719133 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:50.719137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:50.719139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:50.719523 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:50.719804 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:50.720098 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 02:58:50.720327 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:58:50.721235 17453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:50.833161 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 17331
I20260928 02:58:50.850073 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 17463
Sep 28 02:58:50 dist-test-slave-k5rg krb5kdc[17093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 02:58:50.863790 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:50.850740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.12.182.67:33065)
0928 02:58:50.850888 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:50.850893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:50.850980 (+ 87us) server_negotiation.cc:407] Connection header received
0928 02:58:50.851160 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:50.851164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:50.851221 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:50.851300 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:50.852457 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.853174 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.854506 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:50.854889 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:50.854977 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:50.859005 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:50.859015 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:50.859029 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:50.859064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:50.861561 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:50.862197 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:50.862205 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:50.862207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:50.862330 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:50.862600 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:50.862603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:50.862605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:50.862931 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:50.863129 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:50.863374 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:58:50.863543 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:58:50.864459 17585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:50.952823 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 17463
I20260928 02:58:50.965643 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 17106
I20260928 02:58:50.966760 17167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:51.072806 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 17106
2026-09-28T02:58:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4266 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:51Z Disabled control of system clock
I20260928 02:58:51.130637 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:34087
--webserver_interface=127.12.182.126
--webserver_port=0
--builtin_ntp_servers=127.12.182.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:34087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:58:51.433105 17646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:51.433362 17646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:51.433466 17646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:51.440407 17646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:51.440498 17646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:51.440542 17646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:51.440596 17646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:51.440636 17646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:51.449398 17646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:34087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:34087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:51.450626 17646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:51.452261 17646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:51.462985 17651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:51.462978 17652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:51.463017 17654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:51.463963 17646 server_base.cc:1034] running on GCE node
I20260928 02:58:51.464659 17646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:51.465978 17646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:51.467188 17646 hybrid_clock.cc:648] HybridClock initialized: now 1790564331467142 us; error 33 us; skew 500 ppm
I20260928 02:58:51.467495 17646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:58:51.469504 17646 webserver.cc:460] Webserver started at http://127.12.182.126:45913/ using document root <none> and password file <none>
I20260928 02:58:51.470134 17646 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:51.470227 17646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:51.470515 17646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:51.473121 17646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "458ab0f56b8b40628f59ba0e7590227d"
format_stamp: "Formatted at 2026-09-28 02:58:51 on dist-test-slave-k5rg"
server_key: "1fc352b9e7bdc73673a99acca858aa09"
server_key_iv: "300484e4095690d644f05085341f1d5f"
server_key_version: "encryptionkey@0"
I20260928 02:58:51.473860 17646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "458ab0f56b8b40628f59ba0e7590227d"
format_stamp: "Formatted at 2026-09-28 02:58:51 on dist-test-slave-k5rg"
server_key: "1fc352b9e7bdc73673a99acca858aa09"
server_key_iv: "300484e4095690d644f05085341f1d5f"
server_key_version: "encryptionkey@0"
I20260928 02:58:51.478354 17646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:58:51.481402 17660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:51.482571 17646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:58:51.482712 17646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "458ab0f56b8b40628f59ba0e7590227d"
format_stamp: "Formatted at 2026-09-28 02:58:51 on dist-test-slave-k5rg"
server_key: "1fc352b9e7bdc73673a99acca858aa09"
server_key_iv: "300484e4095690d644f05085341f1d5f"
server_key_version: "encryptionkey@0"
I20260928 02:58:51.482890 17646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:51.488133 17646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:51.488831 17646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:51.489099 17646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:51.504602 17646 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:34087
I20260928 02:58:51.504647 17711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:34087 every 8 connection(s)
I20260928 02:58:51.505856 17646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:51.509327 17712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:51.515111 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 17646
I20260928 02:58:51.515259 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:51.515882 13017 external_mini_cluster.cc:1428] Setting key 35e97893cd97ed1c5983b0e682728023
I20260928 02:58:51.519363 17712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d: Bootstrap starting.
I20260928 02:58:51.522305 17712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:51.523222 17712 log.cc:824] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:51.525776 17712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d: No bootstrap required, opened a new log
I20260928 02:58:51.526158 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:51.517759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.12.182.126:34087)
0928 02:58:51.518617 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:51.518627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:51.518663 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:58:51.520217 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:51.520237 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:51.520623 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:51.520962 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:51.524600 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:51.524615 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:51.524627 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:51.524642 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:51.524699 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:51.524715 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:51.524883 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:51.525015 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 02:58:51.525260 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":272,"threads_started":1}
I20260928 02:58:51.533317 17712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ab0f56b8b40628f59ba0e7590227d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34087 } }
I20260928 02:58:51.533671 17712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:51.533756 17712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458ab0f56b8b40628f59ba0e7590227d, State: Initialized, Role: FOLLOWER
I20260928 02:58:51.534293 17712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ab0f56b8b40628f59ba0e7590227d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34087 } }
I20260928 02:58:51.534505 17712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:51.534624 17712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:51.534768 17712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:51.537042 17712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ab0f56b8b40628f59ba0e7590227d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34087 } }
I20260928 02:58:51.537490 17712 leader_election.cc:302] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 458ab0f56b8b40628f59ba0e7590227d; no voters:
I20260928 02:58:51.538069 17712 leader_election.cc:288] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:51.538188 17717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:51.538833 17717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [term 1 LEADER]: Becoming Leader. State: Replica: 458ab0f56b8b40628f59ba0e7590227d, State: Running, Role: LEADER
I20260928 02:58:51.539366 17717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ab0f56b8b40628f59ba0e7590227d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34087 } }
I20260928 02:58:51.539985 17712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:51.542240 17719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 458ab0f56b8b40628f59ba0e7590227d. Latest consensus state: current_term: 1 leader_uuid: "458ab0f56b8b40628f59ba0e7590227d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ab0f56b8b40628f59ba0e7590227d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34087 } } }
I20260928 02:58:51.542627 17719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:51.543195 17718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "458ab0f56b8b40628f59ba0e7590227d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458ab0f56b8b40628f59ba0e7590227d" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34087 } } }
I20260928 02:58:51.543524 17718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:51.546854 17725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:51.551937 17725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:51.560434 17725 catalog_manager.cc:1347] Generated new cluster ID: 21c59b0a5a134138b465b122c2e1e9c7
I20260928 02:58:51.560557 17725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:51.576673 17725 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:51.577064 17725 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:51.585035 17725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d: Generated new TSK 0
I20260928 02:58:51.585887 17725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:58:51.597240 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:34087
--builtin_ntp_servers=127.12.182.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:58:51.935995 17736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:51.936295 17736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:51.936406 17736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:51.943504 17736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:51.943593 17736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:51.943697 17736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:58:51.952457 17736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:34087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:51.953790 17736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:51.955590 17736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:51.968204 17741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:51.968228 17742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:51.968164 17744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:51.969058 17736 server_base.cc:1034] running on GCE node
I20260928 02:58:51.969619 17736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:51.970460 17736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:51.971652 17736 hybrid_clock.cc:648] HybridClock initialized: now 1790564331971606 us; error 38 us; skew 500 ppm
I20260928 02:58:51.971941 17736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:58:51.974149 17736 webserver.cc:460] Webserver started at http://127.12.182.65:44679/ using document root <none> and password file <none>
I20260928 02:58:51.974778 17736 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:51.974862 17736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:51.975167 17736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:51.977891 17736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "1116eacd23224caab97ba99bbb3790ec"
format_stamp: "Formatted at 2026-09-28 02:58:51 on dist-test-slave-k5rg"
server_key: "ed6f879ced96797cc17fc02507e81e40"
server_key_iv: "c9ceb8e349a3ef18fdf0c26d729da1f5"
server_key_version: "encryptionkey@0"
I20260928 02:58:51.978612 17736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "1116eacd23224caab97ba99bbb3790ec"
format_stamp: "Formatted at 2026-09-28 02:58:51 on dist-test-slave-k5rg"
server_key: "ed6f879ced96797cc17fc02507e81e40"
server_key_iv: "c9ceb8e349a3ef18fdf0c26d729da1f5"
server_key_version: "encryptionkey@0"
I20260928 02:58:51.983214 17736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:58:51.986533 17750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:51.987843 17736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:58:51.987982 17736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "1116eacd23224caab97ba99bbb3790ec"
format_stamp: "Formatted at 2026-09-28 02:58:51 on dist-test-slave-k5rg"
server_key: "ed6f879ced96797cc17fc02507e81e40"
server_key_iv: "c9ceb8e349a3ef18fdf0c26d729da1f5"
server_key_version: "encryptionkey@0"
I20260928 02:58:51.988169 17736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:51.993575 17736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:51.994481 17736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:51.994709 17736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:51.995594 17736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:51.996912 17736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:51.996991 17736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:51.997097 17736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:51.997167 17736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:52.026365 17736 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:35183
I20260928 02:58:52.026408 17862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:35183 every 8 connection(s)
I20260928 02:58:52.027645 17736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:58:52.028599 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 17736
I20260928 02:58:52.028784 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:58:52.029541 13017 external_mini_cluster.cc:1428] Setting key c745adb6c7bc5356eb55ea0f2dc2346a
I20260928 02:58:52.032348 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:34087
--builtin_ntp_servers=127.12.182.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:58:52.035763 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.030398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:39087 (local address 127.12.182.126:34087)
0928 02:58:52.030819 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:52.030825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:52.031995 (+ 1170us) server_negotiation.cc:407] Connection header received
0928 02:58:52.033671 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:52.033676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:52.033738 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:52.033828 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:52.035063 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:52.035069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:52.035074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:52.035091 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:52.035131 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:52.035142 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:52.035248 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:52.035415 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 02:58:52.035479 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":707,"threads_started":1}
I20260928 02:58:52.036746 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.030923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34087 (local address 127.12.182.65:39087)
0928 02:58:52.031855 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:52.031891 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:52.033395 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:52.034074 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:52.034082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:52.034172 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:52.034781 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:58:52.034783 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:52.034826 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:58:52.034910 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:52.035224 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:52.035235 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:52.035242 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:58:52.035503 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 02:58:52.035746 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":282,"threads_started":1}
I20260928 02:58:52.038231 17863 heartbeater.cc:347] Connected to a master server at 127.12.182.126:34087
I20260928 02:58:52.038527 17863 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:52.039143 17863 heartbeater.cc:511] Master 127.12.182.126:34087 requested a full tablet report, sending...
I20260928 02:58:52.041157 17677 ts_manager.cc:194] Registered new tserver with Master: 1116eacd23224caab97ba99bbb3790ec (127.12.182.65:35183)
W20260928 02:58:52.337270 17867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:52.337527 17867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:52.337654 17867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:52.344897 17867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:52.344998 17867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:52.345136 17867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:58:52.354146 17867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:34087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:52.355355 17867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:52.357069 17867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:52.369479 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:52.369509 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:52.370141 17867 server_base.cc:1034] running on GCE node
W20260928 02:58:52.370225 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:52.370798 17867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:52.371713 17867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:52.372951 17867 hybrid_clock.cc:648] HybridClock initialized: now 1790564332372901 us; error 35 us; skew 500 ppm
I20260928 02:58:52.373268 17867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:58:52.375396 17867 webserver.cc:460] Webserver started at http://127.12.182.66:44701/ using document root <none> and password file <none>
I20260928 02:58:52.376029 17867 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:52.376123 17867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:52.376438 17867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:52.379266 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "06b9dc8b4c4140ffbb3a193b32f43644"
format_stamp: "Formatted at 2026-09-28 02:58:52 on dist-test-slave-k5rg"
server_key: "00e9bdc1a0ec818c958ed2f7131a8b4f"
server_key_iv: "f1700fef8dbfa2ff472d15f633322b42"
server_key_version: "encryptionkey@0"
I20260928 02:58:52.379992 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "06b9dc8b4c4140ffbb3a193b32f43644"
format_stamp: "Formatted at 2026-09-28 02:58:52 on dist-test-slave-k5rg"
server_key: "00e9bdc1a0ec818c958ed2f7131a8b4f"
server_key_iv: "f1700fef8dbfa2ff472d15f633322b42"
server_key_version: "encryptionkey@0"
I20260928 02:58:52.384666 17867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:58:52.387838 17881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:52.389050 17867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:58:52.389191 17867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "06b9dc8b4c4140ffbb3a193b32f43644"
format_stamp: "Formatted at 2026-09-28 02:58:52 on dist-test-slave-k5rg"
server_key: "00e9bdc1a0ec818c958ed2f7131a8b4f"
server_key_iv: "f1700fef8dbfa2ff472d15f633322b42"
server_key_version: "encryptionkey@0"
I20260928 02:58:52.389384 17867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:52.394654 17867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:52.395355 17867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:52.395588 17867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:52.396413 17867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:52.397739 17867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:52.397828 17867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:52.397931 17867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:52.397997 17867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:52.426182 17867 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:40365
I20260928 02:58:52.426234 17993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:40365 every 8 connection(s)
I20260928 02:58:52.427382 17867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:58:52.430385 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 17867
I20260928 02:58:52.430550 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:58:52.431145 13017 external_mini_cluster.cc:1428] Setting key 2ac397eb8ac6aba6bfa4f8dd3930a165
I20260928 02:58:52.433751 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:34087
--builtin_ntp_servers=127.12.182.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:58:52.435473 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.429858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:46653 (local address 127.12.182.126:34087)
0928 02:58:52.430065 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:52.430070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:52.431608 (+ 1538us) server_negotiation.cc:407] Connection header received
0928 02:58:52.433367 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:52.433372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:52.433446 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:52.433527 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:52.434785 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:52.434791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:52.434796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:52.434815 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:52.434845 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:52.434856 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:52.434967 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:52.435133 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:58:52.435195 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 02:58:52.436303 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.430510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34087 (local address 127.12.182.66:46653)
0928 02:58:52.431465 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:52.431509 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:52.433132 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:52.433767 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:52.433775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:52.433888 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:52.434508 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:58:52.434510 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:52.434546 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:58:52.434628 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:52.434943 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:52.434953 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:52.434961 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:58:52.435196 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 02:58:52.435437 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":294,"threads_started":1}
I20260928 02:58:52.437479 17994 heartbeater.cc:347] Connected to a master server at 127.12.182.126:34087
I20260928 02:58:52.437805 17994 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:52.438439 17994 heartbeater.cc:511] Master 127.12.182.126:34087 requested a full tablet report, sending...
I20260928 02:58:52.439739 17677 ts_manager.cc:194] Registered new tserver with Master: 06b9dc8b4c4140ffbb3a193b32f43644 (127.12.182.66:40365)
W20260928 02:58:52.745517 17997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:52.745786 17997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:52.745913 17997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:52.752959 17997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:52.753046 17997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:52.753180 17997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:58:52.761804 17997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:34087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:52.762979 17997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:52.765255 17997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:52.777421 18003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:52.777457 18005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:52.777387 18002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:52.778563 17997 server_base.cc:1034] running on GCE node
I20260928 02:58:52.779126 17997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:52.779950 17997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:52.781145 17997 hybrid_clock.cc:648] HybridClock initialized: now 1790564332781099 us; error 42 us; skew 500 ppm
I20260928 02:58:52.781424 17997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:58:52.783550 17997 webserver.cc:460] Webserver started at http://127.12.182.67:46471/ using document root <none> and password file <none>
I20260928 02:58:52.784212 17997 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:52.784307 17997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:52.784610 17997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:52.787567 17997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "b153eca3a95e47cfa77dcccb7b86fc04"
format_stamp: "Formatted at 2026-09-28 02:58:52 on dist-test-slave-k5rg"
server_key: "7d07b22fac422413451be5c8c2a4f7e4"
server_key_iv: "b4f687e3e8723f21e43f70c41e4ad589"
server_key_version: "encryptionkey@0"
I20260928 02:58:52.788558 17997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "b153eca3a95e47cfa77dcccb7b86fc04"
format_stamp: "Formatted at 2026-09-28 02:58:52 on dist-test-slave-k5rg"
server_key: "7d07b22fac422413451be5c8c2a4f7e4"
server_key_iv: "b4f687e3e8723f21e43f70c41e4ad589"
server_key_version: "encryptionkey@0"
I20260928 02:58:52.794907 17997 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 02:58:52.798386 18011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:52.799624 17997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:52.799764 17997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "b153eca3a95e47cfa77dcccb7b86fc04"
format_stamp: "Formatted at 2026-09-28 02:58:52 on dist-test-slave-k5rg"
server_key: "7d07b22fac422413451be5c8c2a4f7e4"
server_key_iv: "b4f687e3e8723f21e43f70c41e4ad589"
server_key_version: "encryptionkey@0"
I20260928 02:58:52.799968 17997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:52.804980 17997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:52.805696 17997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:52.805894 17997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:52.806650 17997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:52.807844 17997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:52.807924 17997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:52.807999 17997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:52.808051 17997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:52.834671 17997 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:36411
I20260928 02:58:52.834717 18123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:36411 every 8 connection(s)
I20260928 02:58:52.835883 17997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:58:52.840749 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 17997
I20260928 02:58:52.840886 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:58:52.841341 13017 external_mini_cluster.cc:1428] Setting key 572d980586680e396f31cfe2e88eddce
I20260928 02:58:52.843669 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.838255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:52549 (local address 127.12.182.126:34087)
0928 02:58:52.838422 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:52.838426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:52.839674 (+ 1248us) server_negotiation.cc:407] Connection header received
0928 02:58:52.841501 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:52.841506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:52.841574 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:52.841698 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:52.842960 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:52.842966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:52.842971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:52.842990 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:52.843021 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:52.843032 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:52.843134 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:52.843303 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:58:52.843369 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:58:52.844580 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.838700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34087 (local address 127.12.182.67:52549)
0928 02:58:52.839524 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:52.839558 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:52.841218 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:52.841918 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:52.841926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:52.842031 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:52.842690 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:58:52.842693 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:52.842728 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:58:52.842809 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:52.843111 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:52.843121 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:52.843128 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:58:52.843436 (+ 308us) client_negotiation.cc:240] Negotiation successful
0928 02:58:52.843682 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":247,"threads_started":1}
I20260928 02:58:52.845734 18124 heartbeater.cc:347] Connected to a master server at 127.12.182.126:34087
I20260928 02:58:52.845985 18124 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:52.846518 18124 heartbeater.cc:511] Master 127.12.182.126:34087 requested a full tablet report, sending...
I20260928 02:58:52.847728 17677 ts_manager.cc:194] Registered new tserver with Master: b153eca3a95e47cfa77dcccb7b86fc04 (127.12.182.67:36411)
I20260928 02:58:52.856099 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:58:52.862383 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.860286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.12.182.126:34087)
0928 02:58:52.860465 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:52.860470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:52.860908 (+ 438us) server_negotiation.cc:407] Connection header received
0928 02:58:52.861035 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:52.861039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:52.861089 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:52.861161 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:52.861954 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:52.861958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:52.861961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:52.861975 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:52.862003 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:52.862014 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:52.862102 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:52.862151 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:58:52.862192 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:58:52.865868 17677 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:58:52.872287 17677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:58:52.888439 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.883014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35183 (local address 127.0.0.1:41624)
0928 02:58:52.883860 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:52.883906 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:52.884022 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:52.885683 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:52.885690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:52.885772 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:52.886538 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:58:52.886542 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:52.886569 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:58:52.886695 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:52.887719 (+ 1024us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:52.887729 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:52.887741 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:58:52.887887 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 02:58:52.887983 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":434,"threads_started":1}
I20260928 02:58:52.888726 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.883502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.12.182.66:40365)
0928 02:58:52.884563 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:52.884571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:52.885185 (+ 614us) server_negotiation.cc:407] Connection header received
0928 02:58:52.885247 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:52.885253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:52.885438 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:52.885577 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:52.887971 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:52.887978 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:52.887991 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:52.888015 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:52.888066 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:52.888085 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:52.888242 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:52.888323 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:58:52.888429 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":1061,"threads_started":1}
I20260928 02:58:52.888962 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.885044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:36411 (local address 127.0.0.1:47848)
0928 02:58:52.885502 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:52.885523 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:52.885661 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:52.886597 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:52.886601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:52.886623 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:52.887463 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:58:52.887466 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:52.887482 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:58:52.887593 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:52.888385 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:52.888392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:52.888396 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:58:52.888510 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:58:52.888566 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":644,"threads_started":1}
I20260928 02:58:52.889318 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.883296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:40365 (local address 127.0.0.1:55396)
0928 02:58:52.884968 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:52.884989 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:52.885083 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:52.886674 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:52.886678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:52.886705 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:52.887785 (+ 1080us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:58:52.887788 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:52.887803 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:58:52.887841 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:52.888189 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:52.888193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:52.888198 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:58:52.888332 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 02:58:52.888389 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1542,"thread_start_us":1399,"threads_started":1}
I20260928 02:58:52.889484 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.883225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.12.182.65:35183)
0928 02:58:52.885093 (+ 1868us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:52.885101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:52.885129 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:58:52.885197 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:52.885205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:52.885436 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:52.886185 (+ 749us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:52.887484 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:52.887492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:52.887505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:52.887529 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:52.887581 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:52.887601 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:52.889006 (+ 1405us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:52.889102 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:58:52.889193 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1730,"thread_start_us":1617,"threads_started":1}
I20260928 02:58:52.890007 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:52.885389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.12.182.67:36411)
0928 02:58:52.886143 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:52.886150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:52.886184 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:58:52.886243 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:52.886250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:52.886446 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:52.886597 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:52.888169 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:52.888177 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:52.888188 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:52.888203 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:52.888251 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:52.888271 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:52.889337 (+ 1066us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:52.889548 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:58:52.889670 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":467,"threads_started":1}
I20260928 02:58:52.892894 17929 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb3e71c659f48c6ba67223211b5d907 (DEFAULT_TABLE table=test-table [id=3b095e12773e4a1695d27bb54c93d67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:52.892895 17798 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb3e71c659f48c6ba67223211b5d907 (DEFAULT_TABLE table=test-table [id=3b095e12773e4a1695d27bb54c93d67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:52.893035 18059 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb3e71c659f48c6ba67223211b5d907 (DEFAULT_TABLE table=test-table [id=3b095e12773e4a1695d27bb54c93d67c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:58:52.894196 18059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb3e71c659f48c6ba67223211b5d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:52.894238 17798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb3e71c659f48c6ba67223211b5d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:52.894244 17929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb3e71c659f48c6ba67223211b5d907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:52.906235 18142 tablet_bootstrap.cc:492] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: Bootstrap starting.
I20260928 02:58:52.907685 18143 tablet_bootstrap.cc:492] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec: Bootstrap starting.
I20260928 02:58:52.908618 18144 tablet_bootstrap.cc:492] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: Bootstrap starting.
I20260928 02:58:52.909662 18142 tablet_bootstrap.cc:654] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:52.910872 18142 log.cc:824] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:52.911890 18144 tablet_bootstrap.cc:654] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:52.912099 18143 tablet_bootstrap.cc:654] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:52.913009 18144 log.cc:824] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:52.913167 18143 log.cc:824] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:52.913921 18142 tablet_bootstrap.cc:492] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: No bootstrap required, opened a new log
I20260928 02:58:52.914142 18142 ts_tablet_manager.cc:1397] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260928 02:58:52.915454 18143 tablet_bootstrap.cc:492] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec: No bootstrap required, opened a new log
I20260928 02:58:52.915629 18143 ts_tablet_manager.cc:1397] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260928 02:58:52.915763 18144 tablet_bootstrap.cc:492] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: No bootstrap required, opened a new log
I20260928 02:58:52.915994 18144 ts_tablet_manager.cc:1397] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 02:58:52.922380 18143 raft_consensus.cc:348] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 } } peers { permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 } } peers { permanent_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 36411 } }
I20260928 02:58:52.922708 18143 raft_consensus.cc:374] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:52.922793 18143 raft_consensus.cc:729] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1116eacd23224caab97ba99bbb3790ec, State: Initialized, Role: FOLLOWER
I20260928 02:58:52.923316 18143 consensus_queue.cc:260] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 } } peers { permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 } } peers { permanent_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 36411 } }
I20260928 02:58:52.923967 18142 raft_consensus.cc:348] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 } } peers { permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 } } peers { permanent_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 36411 } }
I20260928 02:58:52.924266 17863 heartbeater.cc:503] Master 127.12.182.126:34087 was elected leader, sending a full tablet report...
I20260928 02:58:52.924360 18142 raft_consensus.cc:374] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:52.924458 18142 raft_consensus.cc:729] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b153eca3a95e47cfa77dcccb7b86fc04, State: Initialized, Role: FOLLOWER
I20260928 02:58:52.924871 18143 ts_tablet_manager.cc:1428] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
I20260928 02:58:52.925047 18142 consensus_queue.cc:260] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 } } peers { permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 } } peers { permanent_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 36411 } }
I20260928 02:58:52.925550 18144 raft_consensus.cc:348] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 } } peers { permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 } } peers { permanent_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 36411 } }
I20260928 02:58:52.925980 18144 raft_consensus.cc:374] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:52.926074 18144 raft_consensus.cc:729] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b9dc8b4c4140ffbb3a193b32f43644, State: Initialized, Role: FOLLOWER
I20260928 02:58:52.926404 18124 heartbeater.cc:503] Master 127.12.182.126:34087 was elected leader, sending a full tablet report...
I20260928 02:58:52.926587 18144 consensus_queue.cc:260] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 } } peers { permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 } } peers { permanent_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 36411 } }
I20260928 02:58:52.926842 18142 ts_tablet_manager.cc:1428] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260928 02:58:52.927886 17994 heartbeater.cc:503] Master 127.12.182.126:34087 was elected leader, sending a full tablet report...
I20260928 02:58:52.928368 18144 ts_tablet_manager.cc:1428] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260928 02:58:52.929131 17995 tablet.cc:2369] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:58:53.029699 17864 tablet.cc:2369] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:58:53.087414 18125 tablet.cc:2369] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:58:53.143940 18149 raft_consensus.cc:482] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:58:53.144236 18149 raft_consensus.cc:504] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 } } peers { permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 } } peers { permanent_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 36411 } }
I20260928 02:58:53.145840 18149 leader_election.cc:288] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1116eacd23224caab97ba99bbb3790ec (127.12.182.65:35183), 06b9dc8b4c4140ffbb3a193b32f43644 (127.12.182.66:40365)
I20260928 02:58:53.148355 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.146176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35183 (local address 127.12.182.67:55155)
0928 02:58:53.146476 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:53.146497 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:53.146612 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:53.146997 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:53.147001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:53.147021 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:53.147613 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:58:53.147616 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:53.147630 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:58:53.147672 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:53.147989 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:53.147994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:53.147999 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:58:53.148109 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:58:53.148155 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260928 02:58:53.148424 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.146384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:55155 (local address 127.12.182.65:35183)
0928 02:58:53.146542 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:53.146548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:53.146568 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:58:53.146750 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:53.146756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:53.146819 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:53.146977 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:53.147821 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:53.147827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:53.147831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:53.147850 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:53.147892 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:53.147905 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:53.147997 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:53.148100 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 02:58:53.148176 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:58:53.149044 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.146297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:50375 (local address 127.12.182.66:40365)
0928 02:58:53.146500 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:53.146505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:53.147099 (+ 594us) server_negotiation.cc:407] Connection header received
0928 02:58:53.147255 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:53.147260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:53.147320 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:53.147410 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:53.148384 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:53.148389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:53.148393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:53.148415 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:53.148451 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:53.148467 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:53.148581 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:53.148744 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:58:53.148796 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:58:53.149195 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.146255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:40365 (local address 127.12.182.67:50375)
0928 02:58:53.146996 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:53.147012 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:53.147136 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:53.147505 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:53.147509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:53.147532 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:53.148159 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:58:53.148162 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:53.148177 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:58:53.148217 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:53.148631 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:53.148635 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:53.148641 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:58:53.148742 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:58:53.148788 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":394,"threads_started":1}
I20260928 02:58:53.149318 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb3e71c659f48c6ba67223211b5d907" candidate_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1116eacd23224caab97ba99bbb3790ec" is_pre_election: true
I20260928 02:58:53.149881 17818 raft_consensus.cc:2454] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b153eca3a95e47cfa77dcccb7b86fc04 in term 0.
I20260928 02:58:53.149982 17949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb3e71c659f48c6ba67223211b5d907" candidate_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" is_pre_election: true
I20260928 02:58:53.150406 17949 raft_consensus.cc:2454] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b153eca3a95e47cfa77dcccb7b86fc04 in term 0.
I20260928 02:58:53.150445 18015 leader_election.cc:302] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1116eacd23224caab97ba99bbb3790ec, b153eca3a95e47cfa77dcccb7b86fc04; no voters:
I20260928 02:58:53.150785 18149 raft_consensus.cc:2781] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:58:53.150949 18149 raft_consensus.cc:482] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:58:53.151058 18149 raft_consensus.cc:3037] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:53.153151 18149 raft_consensus.cc:504] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 } } peers { permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 } } peers { permanent_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 36411 } }
I20260928 02:58:53.153704 18149 leader_election.cc:288] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [CANDIDATE]: Term 1 election: Requested vote from peers 1116eacd23224caab97ba99bbb3790ec (127.12.182.65:35183), 06b9dc8b4c4140ffbb3a193b32f43644 (127.12.182.66:40365)
I20260928 02:58:53.154098 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb3e71c659f48c6ba67223211b5d907" candidate_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1116eacd23224caab97ba99bbb3790ec"
I20260928 02:58:53.154125 17949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb3e71c659f48c6ba67223211b5d907" candidate_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b9dc8b4c4140ffbb3a193b32f43644"
I20260928 02:58:53.154274 17949 raft_consensus.cc:3037] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:53.154273 17818 raft_consensus.cc:3037] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:53.156311 17949 raft_consensus.cc:2454] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b153eca3a95e47cfa77dcccb7b86fc04 in term 1.
I20260928 02:58:53.156348 17818 raft_consensus.cc:2454] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b153eca3a95e47cfa77dcccb7b86fc04 in term 1.
I20260928 02:58:53.156679 18014 leader_election.cc:302] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b9dc8b4c4140ffbb3a193b32f43644, b153eca3a95e47cfa77dcccb7b86fc04; no voters:
I20260928 02:58:53.156972 18149 raft_consensus.cc:2781] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:53.157368 18149 raft_consensus.cc:686] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 1 LEADER]: Becoming Leader. State: Replica: b153eca3a95e47cfa77dcccb7b86fc04, State: Running, Role: LEADER
I20260928 02:58:53.157866 18149 consensus_queue.cc:237] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 } } peers { permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 } } peers { permanent_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 36411 } }
I20260928 02:58:53.162549 17677 catalog_manager.cc:5495] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 reported cstate change: term changed from 0 to 1, leader changed from <none> to b153eca3a95e47cfa77dcccb7b86fc04 (127.12.182.67). New cstate: current_term: 1 leader_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b153eca3a95e47cfa77dcccb7b86fc04" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 36411 } health_report { overall_health: HEALTHY } } }
I20260928 02:58:53.181476 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.179543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.12.182.67:36411)
0928 02:58:53.179701 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:53.179707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:53.179727 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:58:53.179991 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:53.179995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:53.180070 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:53.180153 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:53.180924 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:53.180927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:53.180930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:53.180943 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:53.180972 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:53.180983 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:53.181082 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:53.181173 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:58:53.181230 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:58:53.188906 17818 raft_consensus.cc:1261] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec [term 1 FOLLOWER]: Refusing update from remote peer b153eca3a95e47cfa77dcccb7b86fc04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:53.188905 17949 raft_consensus.cc:1261] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644 [term 1 FOLLOWER]: Refusing update from remote peer b153eca3a95e47cfa77dcccb7b86fc04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:53.189731 18152 consensus_queue.cc:1035] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06b9dc8b4c4140ffbb3a193b32f43644" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 40365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:53.190017 18149 consensus_queue.cc:1035] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1116eacd23224caab97ba99bbb3790ec" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 35183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:53.214749 18156 mvcc.cc:205] Tried to move back new op lower bound from 7334151508733214720 to 7334151508618317824. Current Snapshot: MvccSnapshot[applied={T|T < 7334151508733214720}]
I20260928 02:58:53.230659 17675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:53.230887 17675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:58:53.234288 17675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 458ab0f56b8b40628f59ba0e7590227d: Sending DeleteTablet for 3 replicas of tablet 4bb3e71c659f48c6ba67223211b5d907
I20260928 02:58:53.235266 17798 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb3e71c659f48c6ba67223211b5d907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:53 UTC) from {username='slave'} at 127.0.0.1:41624
I20260928 02:58:53.235373 18059 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb3e71c659f48c6ba67223211b5d907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:53 UTC) from {username='slave'} at 127.0.0.1:47848
I20260928 02:58:53.235368 17929 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb3e71c659f48c6ba67223211b5d907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:58:53 UTC) from {username='slave'} at 127.0.0.1:55396
I20260928 02:58:53.237167 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 17736
I20260928 02:58:53.248471 18167 tablet_replica.cc:330] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: stopping tablet replica
I20260928 02:58:53.248890 18167 raft_consensus.cc:2229] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:58:53.249346 18165 tablet_replica.cc:330] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec: stopping tablet replica
I20260928 02:58:53.249545 18167 raft_consensus.cc:2258] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:53.249758 18165 raft_consensus.cc:2229] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:53.250221 18165 raft_consensus.cc:2258] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:53.250842 18166 tablet_replica.cc:330] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: stopping tablet replica
I20260928 02:58:53.251195 18166 raft_consensus.cc:2229] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:58:53.251427 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.238929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.12.182.65:35183)
0928 02:58:53.239092 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:53.239098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:53.248745 (+ 9647us) server_negotiation.cc:407] Connection header received
0928 02:58:53.248922 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:53.248928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:53.249041 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:53.249156 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:53.250586 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:53.250593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:53.250598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:53.250641 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:53.250679 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:53.250695 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:53.250784 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:53.251087 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:58:53.251148 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:58:53.251633 18166 raft_consensus.cc:2258] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:58:53.252632 17858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:53.254033 18166 ts_tablet_manager.cc:1905] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:53.256106 18167 ts_tablet_manager.cc:1905] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:53.261852 18167 ts_tablet_manager.cc:1918] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:58:53.262007 18167 log.cc:1196] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/4bb3e71c659f48c6ba67223211b5d907
I20260928 02:58:53.262524 18167 ts_tablet_manager.cc:1939] T 4bb3e71c659f48c6ba67223211b5d907 P b153eca3a95e47cfa77dcccb7b86fc04: Deleting consensus metadata
I20260928 02:58:53.264261 17663 catalog_manager.cc:4841] TS b153eca3a95e47cfa77dcccb7b86fc04 (127.12.182.67:36411): tablet 4bb3e71c659f48c6ba67223211b5d907 (table test-table [id=3b095e12773e4a1695d27bb54c93d67c]) successfully deleted
I20260928 02:58:53.266860 18166 ts_tablet_manager.cc:1918] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:58:53.266988 18166 log.cc:1196] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/4bb3e71c659f48c6ba67223211b5d907
I20260928 02:58:53.267360 18166 ts_tablet_manager.cc:1939] T 4bb3e71c659f48c6ba67223211b5d907 P 06b9dc8b4c4140ffbb3a193b32f43644: Deleting consensus metadata
I20260928 02:58:53.268633 17663 catalog_manager.cc:4841] TS 06b9dc8b4c4140ffbb3a193b32f43644 (127.12.182.66:40365): tablet 4bb3e71c659f48c6ba67223211b5d907 (table test-table [id=3b095e12773e4a1695d27bb54c93d67c]) successfully deleted
I20260928 02:58:53.252699 18165 ts_tablet_manager.cc:1905] T 4bb3e71c659f48c6ba67223211b5d907 P 1116eacd23224caab97ba99bbb3790ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:58:53.379292 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 17736
I20260928 02:58:53.398552 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 17867
W20260928 02:58:53.398559 17664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.182.65:35183 (error 108)
I20260928 02:58:53.401470 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.400591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35183 (local address 127.0.0.1:41646)
0928 02:58:53.401155 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:53.401278 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:35183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":373,"threads_started":1}
W20260928 02:58:53.401822 17664 catalog_manager.cc:4584] TS 1116eacd23224caab97ba99bbb3790ec (127.12.182.65:35183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bb3e71c659f48c6ba67223211b5d907: Network error: Client connection negotiation failed: client connection to 127.12.182.65:35183: connect: Connection refused (error 111)
I20260928 02:58:53.402031 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.399384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.12.182.66:40365)
0928 02:58:53.399571 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:53.399578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:53.399666 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:58:53.399871 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:53.399877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:53.399983 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:53.400090 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:53.401213 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:53.401219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:53.401224 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:53.401241 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:53.401279 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:53.401294 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:53.401429 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:53.401698 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 02:58:53.401756 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:58:53.402599 17989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:53.455259 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.454802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35183 (local address 127.0.0.1:41656)
0928 02:58:53.455001 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:53.455097 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:35183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:58:53.456298 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.455955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35183 (local address 127.0.0.1:41660)
0928 02:58:53.456093 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:53.456171 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:35183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:58:53.517032 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.516500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35183 (local address 127.0.0.1:41668)
0928 02:58:53.516769 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:53.516863 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:35183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 02:58:53.520063 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.517858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35183 (local address 127.0.0.1:41676)
0928 02:58:53.519755 (+ 1897us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:53.519852 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:35183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1788,"threads_started":1}
I20260928 02:58:53.558030 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 17867
I20260928 02:58:53.577570 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 17997
I20260928 02:58:53.580536 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.578424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.12.182.67:36411)
0928 02:58:53.578613 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:53.578629 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:53.578655 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:58:53.578764 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:53.578770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:53.578853 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:53.578976 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:53.579858 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:53.579864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:53.579869 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:53.579888 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:53.579926 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:53.579942 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:53.580087 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:53.580171 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:58:53.580227 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:58:53.581267 18119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:53.600920 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.600351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35183 (local address 127.0.0.1:41684)
0928 02:58:53.600590 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:53.600713 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:35183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:58:53.602078 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.601584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35183 (local address 127.0.0.1:41698)
0928 02:58:53.601835 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:53.601932 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:35183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260928 02:58:53.697449 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 17997
I20260928 02:58:53.712395 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 17646
I20260928 02:58:53.713516 17707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:53.815789 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:53.815225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:35183 (local address 127.0.0.1:41712)
0928 02:58:53.815418 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:53.815545 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:35183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:58:53.816061 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 17646
2026-09-28T02:58:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2742 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:58:53.851918 13017 mini_oidc.cc:152] Starting JWKS server
I20260928 02:58:53.854424 13017 webserver.cc:460] Webserver started at https://127.0.0.1:43623/ using document root <none> and password file <none>
I20260928 02:58:53.854737 13017 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:58:53.855347 13017 webserver.cc:460] Webserver started at http://127.0.0.1:35225/ using document root <none> and password file <none>
2026-09-28T02:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:53Z Disabled control of system clock
I20260928 02:58:53.877243 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:39023
--webserver_interface=127.12.182.126
--webserver_port=0
--builtin_ntp_servers=127.12.182.84:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:39023
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43623/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/testchainca.cert with env {}
W20260928 02:58:54.178535 18181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:54.178828 18181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:54.178964 18181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:54.185974 18181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:54.186082 18181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:54.186146 18181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:54.186219 18181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:54.186281 18181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:58:54.186363 18181 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:58:54.186414 18181 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43623/jwks.json
I20260928 02:58:54.195868 18181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:39023
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:39023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43623/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:54.197077 18181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:54.198324 18181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:54.208736 18187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:54.208729 18189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:54.209300 18186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:54.209779 18181 server_base.cc:1034] running on GCE node
I20260928 02:58:54.210492 18181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:54.211751 18181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:54.212945 18181 hybrid_clock.cc:648] HybridClock initialized: now 1790564334212899 us; error 39 us; skew 500 ppm
I20260928 02:58:54.213248 18181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:58:54.215335 18181 webserver.cc:460] Webserver started at http://127.12.182.126:44001/ using document root <none> and password file <none>
I20260928 02:58:54.215960 18181 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:54.216060 18181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:54.216333 18181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:54.219130 18181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "b74d3e76aed44974963f3f5c90812c38"
format_stamp: "Formatted at 2026-09-28 02:58:54 on dist-test-slave-k5rg"
server_key: "70d797a8176c0bf0e6e11ce57a132f3c"
server_key_iv: "7e712ef4ce4e2d030476bd24d68fdee1"
server_key_version: "encryptionkey@0"
I20260928 02:58:54.220095 18181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "b74d3e76aed44974963f3f5c90812c38"
format_stamp: "Formatted at 2026-09-28 02:58:54 on dist-test-slave-k5rg"
server_key: "70d797a8176c0bf0e6e11ce57a132f3c"
server_key_iv: "7e712ef4ce4e2d030476bd24d68fdee1"
server_key_version: "encryptionkey@0"
I20260928 02:58:54.225339 18181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:58:54.228493 18195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:54.229879 18181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:58:54.230032 18181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "b74d3e76aed44974963f3f5c90812c38"
format_stamp: "Formatted at 2026-09-28 02:58:54 on dist-test-slave-k5rg"
server_key: "70d797a8176c0bf0e6e11ce57a132f3c"
server_key_iv: "7e712ef4ce4e2d030476bd24d68fdee1"
server_key_version: "encryptionkey@0"
I20260928 02:58:54.230212 18181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:54.241518 18181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:54.242247 18181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:54.242524 18181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:54.258481 18181 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:39023
I20260928 02:58:54.258487 18246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:39023 every 8 connection(s)
I20260928 02:58:54.259678 18181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:54.260833 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 18181
I20260928 02:58:54.261062 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:54.261883 13017 external_mini_cluster.cc:1428] Setting key 5afdbd823d4621dacccb36cf50390516
I20260928 02:58:54.264001 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:54.276335 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:54.263616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.12.182.126:39023)
0928 02:58:54.264604 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:54.264618 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:54.264665 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:58:54.266708 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:54.266734 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:54.267300 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:54.267728 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:54.268224 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:54.270167 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:54.271502 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:54.272432 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:54.272611 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:54.273649 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:54.273659 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:54.273680 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:54.274363 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:54.274462 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:54.274491 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:54.274600 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:54.274821 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 02:58:54.275139 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":322,"threads_started":1}
I20260928 02:58:54.277097 18247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38: Bootstrap starting.
I20260928 02:58:54.279871 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:58:54.280138 18247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:54.281235 18247 log.cc:824] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:54.283751 18247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38: No bootstrap required, opened a new log
I20260928 02:58:54.290318 18247 raft_consensus.cc:348] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74d3e76aed44974963f3f5c90812c38" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 39023 } }
I20260928 02:58:54.290606 18247 raft_consensus.cc:374] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:54.290673 18247 raft_consensus.cc:729] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b74d3e76aed44974963f3f5c90812c38, State: Initialized, Role: FOLLOWER
I20260928 02:58:54.291170 18247 consensus_queue.cc:260] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74d3e76aed44974963f3f5c90812c38" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 39023 } }
I20260928 02:58:54.291366 18247 raft_consensus.cc:388] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:54.291461 18247 raft_consensus.cc:482] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:54.291577 18247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:54.293457 18247 raft_consensus.cc:504] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74d3e76aed44974963f3f5c90812c38" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 39023 } }
I20260928 02:58:54.293895 18247 leader_election.cc:302] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b74d3e76aed44974963f3f5c90812c38; no voters:
I20260928 02:58:54.294374 18247 leader_election.cc:288] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:54.294502 18252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:54.295161 18252 raft_consensus.cc:686] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [term 1 LEADER]: Becoming Leader. State: Replica: b74d3e76aed44974963f3f5c90812c38, State: Running, Role: LEADER
I20260928 02:58:54.295645 18252 consensus_queue.cc:237] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74d3e76aed44974963f3f5c90812c38" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 39023 } }
I20260928 02:58:54.296162 18247 sys_catalog.cc:564] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:54.298471 18253 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b74d3e76aed44974963f3f5c90812c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74d3e76aed44974963f3f5c90812c38" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 39023 } } }
I20260928 02:58:54.298506 18254 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b74d3e76aed44974963f3f5c90812c38. Latest consensus state: current_term: 1 leader_uuid: "b74d3e76aed44974963f3f5c90812c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74d3e76aed44974963f3f5c90812c38" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 39023 } } }
I20260928 02:58:54.298974 18253 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:54.299006 18254 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:54.299760 18261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:54.304028 18261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:54.311322 18261 catalog_manager.cc:1347] Generated new cluster ID: bf01e81c83fd4e1ba34c1fb9ac783c49
I20260928 02:58:54.311419 18261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:54.336815 18261 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:54.338605 18261 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:54.347383 18261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b74d3e76aed44974963f3f5c90812c38: Generated new TSK 0
I20260928 02:58:54.348284 18261 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:58:54.390273 18277 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:58:54.390501 18277 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:58:54.391405 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:54.371788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.12.182.126:39023)
0928 02:58:54.371947 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:54.371953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:54.372407 (+ 454us) server_negotiation.cc:407] Connection header received
0928 02:58:54.372692 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:54.372696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:54.372765 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:54.372877 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:58:54.373414 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:54.374167 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:54.375520 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:54.375955 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:54.376062 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:54.376569 (+ 507us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:58:54.390771 (+ 14202us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:58:54.391104 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55118: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 02:58:54.391541 18250 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55118: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 02:58:54.392940 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:54.371707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:39023 (local address 127.0.0.1:55118)
0928 02:58:54.372304 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:54.372319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:54.372477 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:54.372932 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:54.372937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:54.372948 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:58:54.373265 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:54.373274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:54.374298 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:54.374302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:54.375375 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:54.375387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:54.376107 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:54.376114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:54.376253 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:54.376320 (+ 67us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:58:54.391053 (+ 14733us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:58:54.391377 (+ 324us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.182.126:39023: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":330,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.182.126:39023: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:58:54.394626 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 18181
I20260928 02:58:54.396059 18242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:54.488550 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 18181
2026-09-28T02:58:54Z chronyd exiting
I20260928 02:58:54.534977 13017 test_util.cc:175] -----------------------------------------------
I20260928 02:58:54.535104 13017 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564295078766-13017-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (687 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:58:54.538654 13017 mini_oidc.cc:152] Starting JWKS server
I20260928 02:58:54.540575 13017 webserver.cc:460] Webserver started at https://127.0.0.1:43807/ using document root <none> and password file <none>
I20260928 02:58:54.540844 13017 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:58:54.541448 13017 webserver.cc:460] Webserver started at http://127.0.0.1:44347/ using document root <none> and password file <none>
2026-09-28T02:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:54Z Disabled control of system clock
I20260928 02:58:54.563494 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:36657
--webserver_interface=127.12.182.126
--webserver_port=0
--builtin_ntp_servers=127.12.182.84:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:36657
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43807/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/testchainca.cert with env {}
W20260928 02:58:54.861508 18284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:54.861792 18284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:54.861936 18284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:54.868860 18284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:54.868953 18284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:54.869014 18284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:54.869095 18284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:54.869148 18284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:58:54.869206 18284 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:58:54.869259 18284 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43807/jwks.json
I20260928 02:58:54.878845 18284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:36657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:36657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43807/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:54.880033 18284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:54.881228 18284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:54.891803 18290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:54.891836 18292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:54.892185 18289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:54.892529 18284 server_base.cc:1034] running on GCE node
I20260928 02:58:54.893185 18284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:54.894495 18284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:54.895709 18284 hybrid_clock.cc:648] HybridClock initialized: now 1790564334895654 us; error 39 us; skew 500 ppm
I20260928 02:58:54.896030 18284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:58:54.898227 18284 webserver.cc:460] Webserver started at http://127.12.182.126:36839/ using document root <none> and password file <none>
I20260928 02:58:54.898834 18284 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:54.898922 18284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:54.899175 18284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:54.901916 18284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "d7fc303056fc427687b68e57fbe86db5"
format_stamp: "Formatted at 2026-09-28 02:58:54 on dist-test-slave-k5rg"
server_key: "9eaa98594b43b67c8fd5a288a4db32a4"
server_key_iv: "b924a901d0e9f706de4f09ff7f0add4d"
server_key_version: "encryptionkey@0"
I20260928 02:58:54.902632 18284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "d7fc303056fc427687b68e57fbe86db5"
format_stamp: "Formatted at 2026-09-28 02:58:54 on dist-test-slave-k5rg"
server_key: "9eaa98594b43b67c8fd5a288a4db32a4"
server_key_iv: "b924a901d0e9f706de4f09ff7f0add4d"
server_key_version: "encryptionkey@0"
I20260928 02:58:54.907428 18284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 02:58:54.910585 18298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:54.911850 18284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:58:54.912009 18284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "d7fc303056fc427687b68e57fbe86db5"
format_stamp: "Formatted at 2026-09-28 02:58:54 on dist-test-slave-k5rg"
server_key: "9eaa98594b43b67c8fd5a288a4db32a4"
server_key_iv: "b924a901d0e9f706de4f09ff7f0add4d"
server_key_version: "encryptionkey@0"
I20260928 02:58:54.912191 18284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:54.928447 18284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:54.929164 18284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:54.929455 18284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:54.945286 18284 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:36657
I20260928 02:58:54.945304 18349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:36657 every 8 connection(s)
I20260928 02:58:54.946530 18284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:54.947324 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 18284
I20260928 02:58:54.947513 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:54.948124 13017 external_mini_cluster.cc:1428] Setting key b480b27361699c56a5ff88a28ef1188e
I20260928 02:58:54.950613 18350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:54.962587 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:54.949866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.12.182.126:36657)
0928 02:58:54.950917 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:54.950931 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:54.950979 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:58:54.953106 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:54.953131 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:54.953740 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:54.954146 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:54.954655 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:54.956593 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:54.957811 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:54.958777 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:54.958941 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:54.959957 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:54.959968 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:54.959986 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:54.960681 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:54.960771 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:54.960796 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:54.960909 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:54.961057 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:58:54.961387 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":332,"threads_started":1}
I20260928 02:58:54.963764 18350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5: Bootstrap starting.
I20260928 02:58:54.965927 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:58:54.966995 18350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:54.968127 18350 log.cc:824] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:54.970927 18350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5: No bootstrap required, opened a new log
I20260928 02:58:54.977886 18350 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fc303056fc427687b68e57fbe86db5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36657 } }
I20260928 02:58:54.978181 18350 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:54.978255 18350 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7fc303056fc427687b68e57fbe86db5, State: Initialized, Role: FOLLOWER
I20260928 02:58:54.978694 18350 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fc303056fc427687b68e57fbe86db5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36657 } }
I20260928 02:58:54.978929 18350 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:54.979027 18350 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:54.979147 18350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:54.981062 18350 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fc303056fc427687b68e57fbe86db5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36657 } }
I20260928 02:58:54.981479 18350 leader_election.cc:302] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7fc303056fc427687b68e57fbe86db5; no voters:
I20260928 02:58:54.982013 18350 leader_election.cc:288] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:54.982129 18355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:54.982762 18355 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [term 1 LEADER]: Becoming Leader. State: Replica: d7fc303056fc427687b68e57fbe86db5, State: Running, Role: LEADER
I20260928 02:58:54.983248 18355 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fc303056fc427687b68e57fbe86db5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36657 } }
I20260928 02:58:54.983800 18350 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:54.985785 18357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7fc303056fc427687b68e57fbe86db5. Latest consensus state: current_term: 1 leader_uuid: "d7fc303056fc427687b68e57fbe86db5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fc303056fc427687b68e57fbe86db5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36657 } } }
I20260928 02:58:54.985857 18356 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7fc303056fc427687b68e57fbe86db5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fc303056fc427687b68e57fbe86db5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36657 } } }
I20260928 02:58:54.986189 18357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:54.986318 18356 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:54.987082 18364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:54.991950 18364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:55.000451 18364 catalog_manager.cc:1347] Generated new cluster ID: 704ace3ae2944100878db2750e0bb20c
I20260928 02:58:55.000545 18364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:55.045956 18364 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:55.047195 18364 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:55.057271 18364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7fc303056fc427687b68e57fbe86db5: Generated new TSK 0
I20260928 02:58:55.058130 18364 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:58:55.100629 18380 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:58:55.100742 18380 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:58:55.102043 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:55.081723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.12.182.126:36657)
0928 02:58:55.081901 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:55.081906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:55.082209 (+ 303us) server_negotiation.cc:407] Connection header received
0928 02:58:55.082412 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:55.082421 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:55.082519 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:55.082675 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:58:55.083116 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:55.084252 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:55.085788 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:55.086542 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:55.086678 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:55.087284 (+ 606us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:58:55.101322 (+ 14038us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:58:55.101729 (+ 407us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34688: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 02:58:55.102136 18379 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:55.081555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:36657 (local address 127.0.0.1:34688)
0928 02:58:55.082122 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:55.082137 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:55.082247 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:55.082685 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:55.082688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:55.082698 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:58:55.082963 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:55.082971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:55.084413 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:55.084419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:55.085635 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:55.085650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:55.086737 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:55.086744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:55.086961 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:55.087007 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:58:55.101573 (+ 14566us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:58:55.101810 (+ 237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.182.126:36657: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":326,"threads_started":1}
W20260928 02:58:55.102166 18353 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34688: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 02:58:55.103333 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 18284
I20260928 02:58:55.104413 18345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:55.199836 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 18284
2026-09-28T02:58:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (712 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:58:55.251320 13017 mini_oidc.cc:152] Starting JWKS server
I20260928 02:58:55.253059 13017 webserver.cc:460] Webserver started at https://127.0.0.1:46523/ using document root <none> and password file <none>
I20260928 02:58:55.253252 13017 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:58:55.253862 13017 webserver.cc:460] Webserver started at http://127.0.0.1:42199/ using document root <none> and password file <none>
2026-09-28T02:58:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:55Z Disabled control of system clock
I20260928 02:58:55.277455 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:42803
--webserver_interface=127.12.182.126
--webserver_port=0
--builtin_ntp_servers=127.12.182.84:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:42803
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46523/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:58:55.576450 18387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:55.576713 18387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:55.576795 18387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:55.583721 18387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:55.583814 18387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:55.583868 18387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:55.583940 18387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:55.583978 18387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:58:55.584038 18387 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:58:55.584076 18387 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46523/jwks.json
I20260928 02:58:55.592818 18387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:42803
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:42803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46523/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:55.594010 18387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:55.595677 18387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:55.606210 18395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:55.606284 18392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:55.606261 18393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:55.607101 18387 server_base.cc:1034] running on GCE node
I20260928 02:58:55.607790 18387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:55.609067 18387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:55.610258 18387 hybrid_clock.cc:648] HybridClock initialized: now 1790564335610216 us; error 29 us; skew 500 ppm
I20260928 02:58:55.610576 18387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:58:55.612614 18387 webserver.cc:460] Webserver started at http://127.12.182.126:44249/ using document root <none> and password file <none>
I20260928 02:58:55.613238 18387 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:55.613332 18387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:55.613655 18387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:55.616312 18387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "5401d3c2be0b4378b583d9c33139a708"
format_stamp: "Formatted at 2026-09-28 02:58:55 on dist-test-slave-k5rg"
server_key: "a4883e227b5c74729536f89e23c749dd"
server_key_iv: "9ab0f0ede381c825e7d800c32915b4f4"
server_key_version: "encryptionkey@0"
I20260928 02:58:55.617049 18387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "5401d3c2be0b4378b583d9c33139a708"
format_stamp: "Formatted at 2026-09-28 02:58:55 on dist-test-slave-k5rg"
server_key: "a4883e227b5c74729536f89e23c749dd"
server_key_iv: "9ab0f0ede381c825e7d800c32915b4f4"
server_key_version: "encryptionkey@0"
I20260928 02:58:55.621686 18387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 02:58:55.624792 18401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:55.626045 18387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:58:55.626194 18387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "5401d3c2be0b4378b583d9c33139a708"
format_stamp: "Formatted at 2026-09-28 02:58:55 on dist-test-slave-k5rg"
server_key: "a4883e227b5c74729536f89e23c749dd"
server_key_iv: "9ab0f0ede381c825e7d800c32915b4f4"
server_key_version: "encryptionkey@0"
I20260928 02:58:55.626384 18387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:55.643380 18387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:55.644116 18387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:55.644419 18387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:55.660835 18387 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:42803
I20260928 02:58:55.660851 18452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:42803 every 8 connection(s)
I20260928 02:58:55.662075 18387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:55.662619 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 18387
I20260928 02:58:55.662832 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:55.663582 13017 external_mini_cluster.cc:1428] Setting key 8ea2140851765e58bf1cd2b409ed63f7
I20260928 02:58:55.666664 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:55.675873 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:55.665385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.12.182.126:42803)
0928 02:58:55.666327 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:55.666338 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:55.666378 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:58:55.667892 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:55.667914 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:55.668334 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:55.668676 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:55.669231 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:55.670552 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:55.671791 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:55.672420 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:55.672578 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:55.673716 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:55.673724 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:55.673737 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:55.674284 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:55.674337 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:55.674362 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:55.674476 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:55.674681 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:58:55.674967 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":329,"threads_started":1}
I20260928 02:58:55.677215 18453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708: Bootstrap starting.
I20260928 02:58:55.679382 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:58:55.680276 18453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:55.681241 18453 log.cc:824] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:55.683811 18453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708: No bootstrap required, opened a new log
I20260928 02:58:55.690907 18453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401d3c2be0b4378b583d9c33139a708" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42803 } }
I20260928 02:58:55.691256 18453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:55.691366 18453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5401d3c2be0b4378b583d9c33139a708, State: Initialized, Role: FOLLOWER
I20260928 02:58:55.691869 18453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401d3c2be0b4378b583d9c33139a708" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42803 } }
I20260928 02:58:55.692099 18453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:55.692217 18453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:55.692373 18453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:55.694782 18453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401d3c2be0b4378b583d9c33139a708" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42803 } }
I20260928 02:58:55.695263 18453 leader_election.cc:302] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5401d3c2be0b4378b583d9c33139a708; no voters:
I20260928 02:58:55.695768 18453 leader_election.cc:288] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:55.695904 18458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:55.696560 18458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [term 1 LEADER]: Becoming Leader. State: Replica: 5401d3c2be0b4378b583d9c33139a708, State: Running, Role: LEADER
I20260928 02:58:55.697135 18458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401d3c2be0b4378b583d9c33139a708" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42803 } }
I20260928 02:58:55.697909 18453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:55.699837 18460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5401d3c2be0b4378b583d9c33139a708. Latest consensus state: current_term: 1 leader_uuid: "5401d3c2be0b4378b583d9c33139a708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401d3c2be0b4378b583d9c33139a708" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42803 } } }
I20260928 02:58:55.700224 18460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:55.700383 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5401d3c2be0b4378b583d9c33139a708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401d3c2be0b4378b583d9c33139a708" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42803 } } }
I20260928 02:58:55.700693 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:55.702024 18466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:55.707275 18466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:55.715813 18466 catalog_manager.cc:1347] Generated new cluster ID: 9680bf25b32242e99572379ca5d16a3e
I20260928 02:58:55.715904 18466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:55.724534 18466 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:55.725739 18466 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:55.731804 18466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5401d3c2be0b4378b583d9c33139a708: Generated new TSK 0
I20260928 02:58:55.732539 18466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:58:55.758075 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:55.750098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.12.182.126:42803)
0928 02:58:55.750241 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:55.750245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:55.750642 (+ 397us) server_negotiation.cc:407] Connection header received
0928 02:58:55.750923 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:55.750928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:55.751000 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:55.751118 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:58:55.751623 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:55.752307 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:55.754258 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:55.754736 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:55.754836 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:55.755418 (+ 582us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:58:55.757446 (+ 2028us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46523/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:58:55.757780 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56194: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46523/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 02:58:55.758193 18456 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56194: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46523/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 02:58:55.760169 18482 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:55.749933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42803 (local address 127.0.0.1:56194)
0928 02:58:55.750541 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:55.750561 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:55.750715 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:55.751152 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:55.751157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:55.751170 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:58:55.751474 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:55.751486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:55.752440 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:55.752446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:55.754115 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:55.754130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:55.754927 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:55.754934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:55.755115 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:55.755160 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:58:55.757668 (+ 2508us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46523/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:58:55.759749 (+ 2081us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.182.126:42803: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46523/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":1803,"thread_start_us":307,"threads_started":1}
I20260928 02:58:55.761098 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 18387
I20260928 02:58:55.762115 18448 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 02:58:55.773862 18483 webserver.cc:533] Webserver: sslize failed:
E20260928 02:58:55.773988 18483 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:58:55.851877 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 18387
2026-09-28T02:58:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (651 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:58:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:55Z Disabled control of system clock
I20260928 02:58:55.925704 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:34535
--webserver_interface=127.12.182.126
--webserver_port=0
--builtin_ntp_servers=127.12.182.84:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:34535
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/testchainca.cert with env {}
W20260928 02:58:56.222545 18488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:56.222816 18488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:56.222927 18488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:56.230890 18488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:56.230988 18488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:56.231052 18488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:56.231129 18488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:56.231189 18488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:58:56.231264 18488 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:58:56.231315 18488 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:58:56.240748 18488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:34535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:34535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:56.241959 18488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:56.243153 18488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:56.254055 18493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:56.254055 18494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:56.254087 18496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:56.254837 18488 server_base.cc:1034] running on GCE node
I20260928 02:58:56.255550 18488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:56.256783 18488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:56.257988 18488 hybrid_clock.cc:648] HybridClock initialized: now 1790564336257944 us; error 35 us; skew 500 ppm
I20260928 02:58:56.258298 18488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:58:56.260342 18488 webserver.cc:460] Webserver started at http://127.12.182.126:39319/ using document root <none> and password file <none>
I20260928 02:58:56.260968 18488 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:56.261065 18488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:56.261330 18488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:56.264097 18488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "ada5fff55b244c1a965242742dd0478a"
format_stamp: "Formatted at 2026-09-28 02:58:56 on dist-test-slave-k5rg"
server_key: "2cb757e09247f4594ff52c7fa1f77e83"
server_key_iv: "3acd0d577bce6676945e423cbfe145e5"
server_key_version: "encryptionkey@0"
I20260928 02:58:56.264833 18488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "ada5fff55b244c1a965242742dd0478a"
format_stamp: "Formatted at 2026-09-28 02:58:56 on dist-test-slave-k5rg"
server_key: "2cb757e09247f4594ff52c7fa1f77e83"
server_key_iv: "3acd0d577bce6676945e423cbfe145e5"
server_key_version: "encryptionkey@0"
I20260928 02:58:56.269465 18488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260928 02:58:56.272621 18502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:56.273917 18488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:58:56.274067 18488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "ada5fff55b244c1a965242742dd0478a"
format_stamp: "Formatted at 2026-09-28 02:58:56 on dist-test-slave-k5rg"
server_key: "2cb757e09247f4594ff52c7fa1f77e83"
server_key_iv: "3acd0d577bce6676945e423cbfe145e5"
server_key_version: "encryptionkey@0"
I20260928 02:58:56.274250 18488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:56.293193 18488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:56.293994 18488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:56.294239 18488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:56.310403 18488 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:34535
I20260928 02:58:56.310423 18553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:34535 every 8 connection(s)
I20260928 02:58:56.311578 18488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:56.315061 18554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:56.320565 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 18488
I20260928 02:58:56.320716 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:56.321308 13017 external_mini_cluster.cc:1428] Setting key 069d7dcab86dde7365df06558bdd54a9
I20260928 02:58:56.324855 18554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a: Bootstrap starting.
I20260928 02:58:56.327900 18554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:56.328963 18554 log.cc:824] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:56.331804 18554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a: No bootstrap required, opened a new log
I20260928 02:58:56.334174 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:56.323088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.12.182.126:34535)
0928 02:58:56.323969 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:56.323979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:56.324018 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:58:56.325576 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:56.325597 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:56.326059 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:56.326360 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:56.326963 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:56.328238 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:56.329940 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:56.330626 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:56.330750 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:56.332031 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:56.332038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:56.332051 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:56.332534 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:56.332603 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:56.332627 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:56.332706 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:56.332941 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 02:58:56.333231 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":285,"threads_started":1}
I20260928 02:58:56.337711 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:58:56.339700 18554 raft_consensus.cc:348] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada5fff55b244c1a965242742dd0478a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34535 } }
I20260928 02:58:56.340070 18554 raft_consensus.cc:374] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:56.340184 18554 raft_consensus.cc:729] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5fff55b244c1a965242742dd0478a, State: Initialized, Role: FOLLOWER
I20260928 02:58:56.340761 18554 consensus_queue.cc:260] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada5fff55b244c1a965242742dd0478a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34535 } }
I20260928 02:58:56.340986 18554 raft_consensus.cc:388] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:56.341089 18554 raft_consensus.cc:482] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:56.341203 18554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:56.343273 18554 raft_consensus.cc:504] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada5fff55b244c1a965242742dd0478a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34535 } }
I20260928 02:58:56.343773 18554 leader_election.cc:302] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ada5fff55b244c1a965242742dd0478a; no voters:
I20260928 02:58:56.344363 18554 leader_election.cc:288] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:56.344501 18559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:56.345085 18559 raft_consensus.cc:686] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [term 1 LEADER]: Becoming Leader. State: Replica: ada5fff55b244c1a965242742dd0478a, State: Running, Role: LEADER
I20260928 02:58:56.345561 18559 consensus_queue.cc:237] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada5fff55b244c1a965242742dd0478a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34535 } }
I20260928 02:58:56.346261 18554 sys_catalog.cc:564] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:56.348237 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ada5fff55b244c1a965242742dd0478a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada5fff55b244c1a965242742dd0478a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34535 } } }
I20260928 02:58:56.348253 18561 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada5fff55b244c1a965242742dd0478a. Latest consensus state: current_term: 1 leader_uuid: "ada5fff55b244c1a965242742dd0478a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada5fff55b244c1a965242742dd0478a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 34535 } } }
I20260928 02:58:56.348690 18561 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:56.349691 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:56.350176 18566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:56.354619 18566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:56.364401 18566 catalog_manager.cc:1347] Generated new cluster ID: 642198b1ea8243f8965af6975da96666
I20260928 02:58:56.364502 18566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:56.374204 18566 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:56.375900 18566 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:56.389552 18566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ada5fff55b244c1a965242742dd0478a: Generated new TSK 0
I20260928 02:58:56.390322 18566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:58:56.826061 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:56.403963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.12.182.126:34535)
0928 02:58:56.404132 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:56.404137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:56.404424 (+ 287us) server_negotiation.cc:407] Connection header received
0928 02:58:56.404612 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:56.404618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:56.404702 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:56.404853 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:58:56.405220 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:56.406332 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:56.407649 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:56.408321 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:56.408459 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:56.408931 (+ 472us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:58:56.825503 (+416572us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:58:56.825832 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58332: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 02:58:56.826167 18557 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58332: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 02:58:56.826376 18583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:56.403812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:34535 (local address 127.0.0.1:58332)
0928 02:58:56.404329 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:56.404343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:56.404450 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:56.404847 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:56.404850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:56.404858 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:58:56.405093 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:56.405101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:56.406459 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:56.406463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:56.407514 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:56.407524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:56.408506 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:56.408511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:56.408645 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:56.408679 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:58:56.825786 (+417107us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:58:56.826031 (+ 245us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.182.126:34535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":257,"threads_started":1}
I20260928 02:58:56.827593 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 18488
I20260928 02:58:56.828588 18549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:56.927989 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 18488
2026-09-28T02:58:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1058 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (379 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (383 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:57 dist-test-slave-k5rg krb5kdc[18598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:57 dist-test-slave-k5rg krb5kdc[18598](info): set up 2 sockets
Sep 28 02:58:57 dist-test-slave-k5rg krb5kdc[18598](info): commencing operation
krb5kdc: starting...
W20260928 02:58:58.710062 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:58:58 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:58:58.915078 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:36797
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:36797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:59.232458 18611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:59.232739 18611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:59.232877 18611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:59.240854 18611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:59.240998 18611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:59.241076 18611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:59.241151 18611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:59.241209 18611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:59.250766 18611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:36797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:36797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:59.251969 18611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:59.253194 18611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:59.268697 18616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:59.268731 18617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:59.268666 18619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:59.269641 18611 server_base.cc:1034] running on GCE node
I20260928 02:58:59.270398 18611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:59.271759 18611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:59.272958 18611 hybrid_clock.cc:648] HybridClock initialized: now 1790564339272896 us; error 51 us; skew 500 ppm
I20260928 02:58:59.273276 18611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:59 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564339, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:59.276170 18611 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:58:59.277495 18611 webserver.cc:460] Webserver started at http://127.12.182.126:43985/ using document root <none> and password file <none>
I20260928 02:58:59.278144 18611 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:59.278240 18611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:59.278498 18611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:59.281222 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "d37306b52eea490aa90cad4eeb63c2fe"
format_stamp: "Formatted at 2026-09-28 02:58:59 on dist-test-slave-k5rg"
server_key: "ac27937ae3f5376313ebd21658a1180b"
server_key_iv: "7150e92ea4b84293a9f922d6d291d3ef"
server_key_version: "encryptionkey@0"
I20260928 02:58:59.281986 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "d37306b52eea490aa90cad4eeb63c2fe"
format_stamp: "Formatted at 2026-09-28 02:58:59 on dist-test-slave-k5rg"
server_key: "ac27937ae3f5376313ebd21658a1180b"
server_key_iv: "7150e92ea4b84293a9f922d6d291d3ef"
server_key_version: "encryptionkey@0"
I20260928 02:58:59.286769 18611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:58:59.290057 18626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:59.291272 18611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:59.291437 18611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "d37306b52eea490aa90cad4eeb63c2fe"
format_stamp: "Formatted at 2026-09-28 02:58:59 on dist-test-slave-k5rg"
server_key: "ac27937ae3f5376313ebd21658a1180b"
server_key_iv: "7150e92ea4b84293a9f922d6d291d3ef"
server_key_version: "encryptionkey@0"
I20260928 02:58:59.291615 18611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:59.322736 18611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:59.327011 18611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:59.327275 18611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:59.343765 18611 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:36797
I20260928 02:58:59.343794 18677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:36797 every 8 connection(s)
I20260928 02:58:59.345049 18611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:59.345852 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 18611
I20260928 02:58:59.346051 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:58:59.346710 13017 external_mini_cluster.cc:1428] Setting key 860db950c9df1d4939c1f83c728b3221
I20260928 02:58:59.349107 18678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:58:59 dist-test-slave-k5rg krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:58:59.361423 18678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe: Bootstrap starting.
I20260928 02:58:59.364488 18678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:59.365563 18678 log.cc:824] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:59.368556 18678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe: No bootstrap required, opened a new log
I20260928 02:58:59.368947 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:59.348550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.12.182.126:36797)
0928 02:58:59.349333 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:59.349348 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:59.349406 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:58:59.351668 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:59.351694 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:59.352236 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:59.352650 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:59.353865 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:59.355817 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:59.357003 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:59.357991 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:59.358161 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:59.362156 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:59.362164 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:59.362178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:59.362212 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:59.365011 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:59.365771 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:59.365777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:59.365783 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:59.365932 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:59.366378 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:59.366383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:59.366386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:59.367000 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:59.367204 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:59.367649 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 02:58:59.367959 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":307,"threads_started":1}
I20260928 02:58:59.376348 18678 raft_consensus.cc:348] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37306b52eea490aa90cad4eeb63c2fe" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36797 } }
I20260928 02:58:59.376696 18678 raft_consensus.cc:374] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:59.376789 18678 raft_consensus.cc:729] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37306b52eea490aa90cad4eeb63c2fe, State: Initialized, Role: FOLLOWER
I20260928 02:58:59.377290 18678 consensus_queue.cc:260] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37306b52eea490aa90cad4eeb63c2fe" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36797 } }
I20260928 02:58:59.377508 18678 raft_consensus.cc:388] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:59.377655 18678 raft_consensus.cc:482] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:59.377807 18678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:59.379882 18678 raft_consensus.cc:504] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37306b52eea490aa90cad4eeb63c2fe" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36797 } }
I20260928 02:58:59.380344 18678 leader_election.cc:302] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d37306b52eea490aa90cad4eeb63c2fe; no voters:
I20260928 02:58:59.380890 18678 leader_election.cc:288] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:59.381027 18683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:59.381596 18683 raft_consensus.cc:686] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [term 1 LEADER]: Becoming Leader. State: Replica: d37306b52eea490aa90cad4eeb63c2fe, State: Running, Role: LEADER
I20260928 02:58:59.382148 18683 consensus_queue.cc:237] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37306b52eea490aa90cad4eeb63c2fe" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36797 } }
I20260928 02:58:59.382710 18678 sys_catalog.cc:564] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:59.384644 18685 sys_catalog.cc:455] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d37306b52eea490aa90cad4eeb63c2fe. Latest consensus state: current_term: 1 leader_uuid: "d37306b52eea490aa90cad4eeb63c2fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37306b52eea490aa90cad4eeb63c2fe" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36797 } } }
I20260928 02:58:59.385051 18685 sys_catalog.cc:458] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:59.385028 18684 sys_catalog.cc:455] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d37306b52eea490aa90cad4eeb63c2fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37306b52eea490aa90cad4eeb63c2fe" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 36797 } } }
I20260928 02:58:59.385480 18684 sys_catalog.cc:458] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:59.390241 18692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:59.395445 18692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:59.403590 18692 catalog_manager.cc:1347] Generated new cluster ID: 4fb8818c82db407894fee49d2bc53422
I20260928 02:58:59.403704 18692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:59.424083 18692 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:59.425379 18692 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:59.432866 18692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d37306b52eea490aa90cad4eeb63c2fe: Generated new TSK 0
I20260928 02:58:59.433579 18692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:58:59.488595 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:36797
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:59.793277 18711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:59.793540 18711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:59.793696 18711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:59.800647 18711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:59.800760 18711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:59.800922 18711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:58:59.810304 18711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:36797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:59.811497 18711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:59.812733 18711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:59.824651 18716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:59.824841 18717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:59.824859 18719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:59.825420 18711 server_base.cc:1034] running on GCE node
I20260928 02:58:59.825984 18711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:59.826867 18711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:59.828053 18711 hybrid_clock.cc:648] HybridClock initialized: now 1790564339828012 us; error 37 us; skew 500 ppm
I20260928 02:58:59.828327 18711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:59 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564339, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:59.833829 18711 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:58:59.835189 18711 webserver.cc:460] Webserver started at http://127.12.182.65:42221/ using document root <none> and password file <none>
I20260928 02:58:59.835810 18711 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:59.835906 18711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:59.836138 18711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:59.838881 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "c309c3e5cd0942d9aa3ae96c0c58651f"
format_stamp: "Formatted at 2026-09-28 02:58:59 on dist-test-slave-k5rg"
server_key: "5a622cd54ca8f7141b782e20d085c601"
server_key_iv: "bac0992e601b4f8991add3b85bc91e83"
server_key_version: "encryptionkey@0"
I20260928 02:58:59.839600 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "c309c3e5cd0942d9aa3ae96c0c58651f"
format_stamp: "Formatted at 2026-09-28 02:58:59 on dist-test-slave-k5rg"
server_key: "5a622cd54ca8f7141b782e20d085c601"
server_key_iv: "bac0992e601b4f8991add3b85bc91e83"
server_key_version: "encryptionkey@0"
I20260928 02:58:59.844430 18711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:58:59.847694 18726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:59.848922 18711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:58:59.849040 18711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "c309c3e5cd0942d9aa3ae96c0c58651f"
format_stamp: "Formatted at 2026-09-28 02:58:59 on dist-test-slave-k5rg"
server_key: "5a622cd54ca8f7141b782e20d085c601"
server_key_iv: "bac0992e601b4f8991add3b85bc91e83"
server_key_version: "encryptionkey@0"
I20260928 02:58:59.849197 18711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:59.862790 18711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:59.866864 18711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:59.867116 18711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:59.867959 18711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:58:59.869197 18711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:58:59.869266 18711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:59.869382 18711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:58:59.869436 18711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:59.905100 18711 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:42309
I20260928 02:58:59.905169 18838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:42309 every 8 connection(s)
I20260928 02:58:59.906628 18711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:58:59.907454 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 18711
I20260928 02:58:59.907627 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:58:59.908396 13017 external_mini_cluster.cc:1428] Setting key 704806ff6682dd3e3152040afaafec2b
Sep 28 02:58:59 dist-test-slave-k5rg krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564339, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
I20260928 02:58:59.940618 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:59.910063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:36797 (local address 127.12.182.65:32771)
0928 02:58:59.919345 (+ 9282us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:59.919393 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:59.921333 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:59.922244 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:59.922257 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:59.922969 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:59.923728 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:59.923753 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:59.925210 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:59.925216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:59.927133 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:59.927148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:59.928317 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:59.928323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:59.928607 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:59.929312 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:59.929339 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:59.932108 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:59.935849 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:59.935858 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:59.935876 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:59.936478 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:59.936953 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:59.936958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:59.936961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:59.937216 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:59.937997 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:59.938005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:59.938374 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 02:58:59.938675 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 02:58:59.939223 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9009,"thread_start_us":8821,"threads_started":1}
I20260928 02:58:59.941552 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:59.909393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:32771 (local address 127.12.182.126:36797)
0928 02:58:59.920080 (+ 10687us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:59.920086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:59.920106 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:58:59.921634 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:59.921639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:59.921728 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:59.921825 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:59.923952 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:59.925046 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:59.927305 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:59.927921 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:59.928029 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:59.932341 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:59.932348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:59.932353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:59.932391 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:59.935679 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:59.936635 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:59.936642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:59.936645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:59.936795 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:59.937358 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:59.937364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:59.937367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:59.937754 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:59.937925 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:59.941110 (+ 3185us) server_negotiation.cc:299] Negotiation successful
0928 02:58:59.941317 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10563,"thread_start_us":10466,"threads_started":1}
I20260928 02:58:59.942762 18839 heartbeater.cc:347] Connected to a master server at 127.12.182.126:36797
I20260928 02:58:59.943104 18839 heartbeater.cc:464] Registering TS with master...
I20260928 02:58:59.943949 18839 heartbeater.cc:511] Master 127.12.182.126:36797 requested a full tablet report, sending...
I20260928 02:58:59.946167 18643 ts_manager.cc:194] Registered new tserver with Master: c309c3e5cd0942d9aa3ae96c0c58651f (127.12.182.65:42309)
I20260928 02:58:59.947741 18643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:32771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:58:59.964450 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:36797
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:00.265014 18845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:00.265264 18845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:00.265363 18845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:00.272264 18845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:00.272356 18845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:00.272470 18845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:59:00.281622 18845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:36797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:00.282774 18845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:00.284096 18845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:00.296057 18850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:00.296018 18853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:00.296094 18851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:00.296968 18845 server_base.cc:1034] running on GCE node
I20260928 02:59:00.297533 18845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:00.298450 18845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:00.299649 18845 hybrid_clock.cc:648] HybridClock initialized: now 1790564340299592 us; error 47 us; skew 500 ppm
I20260928 02:59:00.299924 18845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:00.302727 18845 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:59:00.304013 18845 webserver.cc:460] Webserver started at http://127.12.182.66:41123/ using document root <none> and password file <none>
I20260928 02:59:00.304616 18845 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:00.304724 18845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:00.304998 18845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:00.307693 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "f0dbf479d0104dbcb03b2566ccbb95d2"
format_stamp: "Formatted at 2026-09-28 02:59:00 on dist-test-slave-k5rg"
server_key: "d4a95b7c3730643bbc405594a3fe3900"
server_key_iv: "66e6f007a9fdb39618e0a01b25539217"
server_key_version: "encryptionkey@0"
I20260928 02:59:00.308391 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "f0dbf479d0104dbcb03b2566ccbb95d2"
format_stamp: "Formatted at 2026-09-28 02:59:00 on dist-test-slave-k5rg"
server_key: "d4a95b7c3730643bbc405594a3fe3900"
server_key_iv: "66e6f007a9fdb39618e0a01b25539217"
server_key_version: "encryptionkey@0"
I20260928 02:59:00.312999 18845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:59:00.316336 18860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:00.317518 18845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:59:00.317673 18845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "f0dbf479d0104dbcb03b2566ccbb95d2"
format_stamp: "Formatted at 2026-09-28 02:59:00 on dist-test-slave-k5rg"
server_key: "d4a95b7c3730643bbc405594a3fe3900"
server_key_iv: "66e6f007a9fdb39618e0a01b25539217"
server_key_version: "encryptionkey@0"
I20260928 02:59:00.317843 18845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:00.340986 18845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:00.344841 18845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:00.345094 18845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:00.346123 18845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:00.347422 18845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:00.347499 18845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:00.347609 18845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:00.347659 18845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:00.378486 18845 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:39787
I20260928 02:59:00.378551 18973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:39787 every 8 connection(s)
I20260928 02:59:00.379666 18845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:59:00.380203 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 18845
I20260928 02:59:00.380354 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:59:00.380887 13017 external_mini_cluster.cc:1428] Setting key fe8371561d1a4e11966a7fbe89d4132a
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
I20260928 02:59:00.399127 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:00.382232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:53467 (local address 127.12.182.126:36797)
0928 02:59:00.382383 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:00.382388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:00.383874 (+ 1486us) server_negotiation.cc:407] Connection header received
0928 02:59:00.385784 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:00.385789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:00.385864 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:00.385954 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:00.387359 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.388077 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.389533 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.390020 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.390122 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:00.393495 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:00.393500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:00.393503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:00.393533 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:00.395935 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.396806 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.396810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.396812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.396960 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.397444 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.397449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.397453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.397776 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:00.397919 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:00.398618 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 02:59:00.398813 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:59:00.399978 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:00.382863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:36797 (local address 127.12.182.66:53467)
0928 02:59:00.383694 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:00.383745 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:00.385502 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:00.386174 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:00.386182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:00.386670 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:00.387203 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:00.387221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.388215 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.388219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:00.389394 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:00.389406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.390231 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.390239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:00.390453 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:00.391048 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:00.391069 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:00.393281 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:00.396061 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:00.396071 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:00.396088 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:00.396674 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:00.397079 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:00.397084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:00.397088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:00.397327 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:00.397907 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:00.397915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:00.398302 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 02:59:00.398533 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 02:59:00.398921 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":321,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
I20260928 02:59:00.401337 18974 heartbeater.cc:347] Connected to a master server at 127.12.182.126:36797
I20260928 02:59:00.401685 18974 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:00.402452 18974 heartbeater.cc:511] Master 127.12.182.126:36797 requested a full tablet report, sending...
I20260928 02:59:00.403950 18643 ts_manager.cc:194] Registered new tserver with Master: f0dbf479d0104dbcb03b2566ccbb95d2 (127.12.182.66:39787)
I20260928 02:59:00.404731 18643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:53467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:59:00.421502 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:36797
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:00.720355 18984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:00.720599 18984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:00.720690 18984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:00.727674 18984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:00.727771 18984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:00.727876 18984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:59:00.737077 18984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:36797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:00.738234 18984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:00.739418 18984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:00.751292 18989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:00.751356 18990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:00.751740 18992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:00.752236 18984 server_base.cc:1034] running on GCE node
I20260928 02:59:00.752878 18984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:00.753803 18984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:00.754987 18984 hybrid_clock.cc:648] HybridClock initialized: now 1790564340754941 us; error 44 us; skew 500 ppm
I20260928 02:59:00.755256 18984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:00.758121 18984 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:59:00.759459 18984 webserver.cc:460] Webserver started at http://127.12.182.67:33381/ using document root <none> and password file <none>
I20260928 02:59:00.760076 18984 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:00.760180 18984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:00.760460 18984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:00.763180 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "c2867799f09c4b53954ec3efc96b5798"
format_stamp: "Formatted at 2026-09-28 02:59:00 on dist-test-slave-k5rg"
server_key: "66d2012e0ea730fd4e27ceb8431e7ebd"
server_key_iv: "f77ba6e444f9d5a3372c4c76395b00a7"
server_key_version: "encryptionkey@0"
I20260928 02:59:00.763922 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "c2867799f09c4b53954ec3efc96b5798"
format_stamp: "Formatted at 2026-09-28 02:59:00 on dist-test-slave-k5rg"
server_key: "66d2012e0ea730fd4e27ceb8431e7ebd"
server_key_iv: "f77ba6e444f9d5a3372c4c76395b00a7"
server_key_version: "encryptionkey@0"
I20260928 02:59:00.768970 18984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:59:00.772287 18999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:00.773540 18984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:59:00.773712 18984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "c2867799f09c4b53954ec3efc96b5798"
format_stamp: "Formatted at 2026-09-28 02:59:00 on dist-test-slave-k5rg"
server_key: "66d2012e0ea730fd4e27ceb8431e7ebd"
server_key_iv: "f77ba6e444f9d5a3372c4c76395b00a7"
server_key_version: "encryptionkey@0"
I20260928 02:59:00.773895 18984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:00.785986 18984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:00.794134 18984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:00.794364 18984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:00.795197 18984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:00.796437 18984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:00.796506 18984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:00.796593 18984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:00.796636 18984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:00.823583 18984 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:41231
I20260928 02:59:00.823630 19111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:41231 every 8 connection(s)
I20260928 02:59:00.824748 18984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:59:00.827159 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 18984
I20260928 02:59:00.827322 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:59:00.827780 13017 external_mini_cluster.cc:1428] Setting key 4cf82b04248d1ad7640de49269345497
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:59:00.843087 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:00.827212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:40737 (local address 127.12.182.126:36797)
0928 02:59:00.827421 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:00.827427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:00.828640 (+ 1213us) server_negotiation.cc:407] Connection header received
0928 02:59:00.830465 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:00.830471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:00.830551 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:00.830634 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:00.832064 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.832766 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.834274 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.834689 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.834795 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:00.837979 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:00.837983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:00.837986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:00.838017 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:00.840327 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.840961 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.840966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.840968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.841088 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.841476 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.841479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.841481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.841779 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:00.841919 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:00.842549 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 02:59:00.842753 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:59:00.843950 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:00.827676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:36797 (local address 127.12.182.67:40737)
0928 02:59:00.828479 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:00.828532 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:00.830210 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:00.830867 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:00.830874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:00.831352 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:00.831902 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:00.831919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.832905 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.832909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:00.834144 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:00.834156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.834889 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.834896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:00.835119 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:00.835703 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:00.835724 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:00.837792 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:00.840452 (+ 2660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:00.840458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:00.840469 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:00.840831 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:00.841187 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:00.841191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:00.841194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:00.841367 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:00.841934 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:00.841940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:00.842220 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 02:59:00.842432 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 02:59:00.842909 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":351,"threads_started":1}
I20260928 02:59:00.845031 19112 heartbeater.cc:347] Connected to a master server at 127.12.182.126:36797
I20260928 02:59:00.845299 19112 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:00.845983 19112 heartbeater.cc:511] Master 127.12.182.126:36797 requested a full tablet report, sending...
I20260928 02:59:00.847195 18643 ts_manager.cc:194] Registered new tserver with Master: c2867799f09c4b53954ec3efc96b5798 (127.12.182.67:41231)
I20260928 02:59:00.847936 18643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:40737
I20260928 02:59:00.854245 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/test-admin.keytab.
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:00.878657 13017 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:59:00.883078 19119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:00.880899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.12.182.65:42309)
0928 02:59:00.881441 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:00.881449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:00.881478 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:59:00.881653 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:00.881660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:00.881866 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:00.882018 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:00.882746 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56080: BlockingRecv error: recv got EOF from 127.0.0.1:56080 (error 108)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":305,"threads_started":1}
W20260928 02:59:00.883163 13017 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:00.884555 13017 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:00.885852 13017 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:59:00.899806 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:00.887231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.12.182.65:42309)
0928 02:59:00.887411 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:00.887417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:00.887614 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:59:00.887728 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:00.887731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:00.887785 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:00.887858 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:00.888784 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.889801 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.890935 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.891479 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.891570 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:00.894755 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:00.894766 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:00.894779 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:00.894809 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:00.897523 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.898125 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.898133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.898135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.898251 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.898531 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.898534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.898536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.898840 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:00.899052 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:00.899333 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:59:00.899562 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 02:59:00.904956 19119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:00.903303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56106 (local address 127.12.182.65:42309)
0928 02:59:00.903442 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:00.903445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:00.903724 (+ 279us) server_negotiation.cc:407] Connection header received
0928 02:59:00.903831 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:00.903835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:00.903906 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:00.903999 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:00.904720 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56106: BlockingRecv error: recv got EOF from 127.0.0.1:56106 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 02:59:00.905089 13017 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:00.906706 13017 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:00.908118 13017 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:59:00.921730 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:00.909431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.12.182.65:42309)
0928 02:59:00.909576 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:00.909581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:00.909886 (+ 305us) server_negotiation.cc:407] Connection header received
0928 02:59:00.910118 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:00.910123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:00.910212 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:00.910302 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:00.911487 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.912185 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.913831 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.914257 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.914342 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:00.917379 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:00.917383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:00.917386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:00.917427 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:00.919578 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.920121 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.920125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.920127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.920227 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.920480 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.920483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.920485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.920732 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:00.920961 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:00.921250 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:59:00.921446 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 02:59:00.925230 19119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:00.923789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.12.182.65:42309)
0928 02:59:00.923978 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:00.923983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:00.924170 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:59:00.924262 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:00.924266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:00.924323 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:00.924390 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:00.925035 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56130: BlockingRecv error: recv got EOF from 127.0.0.1:56130 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:59:00.925350 13017 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564295078766-13017-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:00.926656 13017 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:00.927769 13017 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:59:00 dist-test-slave-k5rg krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:59:00.939673 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:00.928933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.12.182.65:42309)
0928 02:59:00.929094 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:00.929099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:00.929226 (+ 127us) server_negotiation.cc:407] Connection header received
0928 02:59:00.929358 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:00.929362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:00.929409 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:00.929481 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:00.930345 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.931008 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.932156 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.932536 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.932635 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:00.935632 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:00.935636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:00.935639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:00.935665 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:00.937871 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.938349 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.938352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.938355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.938464 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.938707 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.938710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.938712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.938925 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:00.939139 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:00.939306 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 02:59:00.939445 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:59:00.940949 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 18711
I20260928 02:59:00.950511 18839 heartbeater.cc:503] Master 127.12.182.126:36797 was elected leader, sending a full tablet report...
I20260928 02:59:00.950842 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:00.941562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.12.182.65:42309)
0928 02:59:00.941730 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:00.941735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:00.941924 (+ 189us) server_negotiation.cc:407] Connection header received
0928 02:59:00.942027 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:00.942030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:00.942074 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:00.942139 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:00.942881 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.943501 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.944563 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:00.944933 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:00.945031 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:00.946740 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:00.946744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:00.946747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:00.946773 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:00.948764 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.949279 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.949282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.949284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.949388 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:00.949688 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:00.949691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:00.949693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:00.949904 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:00.950080 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:00.950368 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:59:00.950556 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:59:00.951361 18834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:01.032155 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 18711
I20260928 02:59:01.044826 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 18845
Sep 28 02:59:01 dist-test-slave-k5rg krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 02:59:01.059007 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:01.045573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.12.182.66:39787)
0928 02:59:01.046122 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:01.046129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:01.046153 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:59:01.046209 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:01.046215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:01.046406 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:01.046536 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:01.047389 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:01.048351 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:01.049532 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:01.050167 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:01.050262 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:01.053717 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:01.053728 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:01.053740 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:01.053778 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:01.056353 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:01.056980 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:01.056987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:01.056989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:01.057103 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:01.057443 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:01.057448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:01.057450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:01.057888 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:01.058177 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:01.058480 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:59:01.058717 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":265,"threads_started":1}
I20260928 02:59:01.060220 18969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:01.141656 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 18845
I20260928 02:59:01.154724 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 18984
Sep 28 02:59:01 dist-test-slave-k5rg krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 02:59:01.169548 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:01.155757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.12.182.67:41231)
0928 02:59:01.156402 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:01.156409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:01.156438 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:59:01.156513 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:01.156519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:01.156728 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:01.156848 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:01.157837 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:01.158826 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:01.159962 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:01.160508 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:01.160600 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:01.164205 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:01.164216 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:01.164230 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:01.164261 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:01.166823 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:01.167472 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:01.167479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:01.167481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:01.167594 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:01.167955 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:01.167958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:01.167960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:01.168284 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:01.168649 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:01.168878 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 02:59:01.169191 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":345,"threads_started":1}
I20260928 02:59:01.170928 19107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:01.254257 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 18984
I20260928 02:59:01.266489 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 18611
I20260928 02:59:01.267669 18673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:01.368783 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 18611
2026-09-28T02:59:01Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3680 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:59:01 dist-test-slave-k5rg krb5kdc[19139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:59:01 dist-test-slave-k5rg krb5kdc[19139](info): set up 2 sockets
Sep 28 02:59:01 dist-test-slave-k5rg krb5kdc[19139](info): commencing operation
krb5kdc: starting...
W20260928 02:59:02.358103 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:59:02 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:59:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.126.keytab.
Entry for principal oryx/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.126.keytab.
I20260928 02:59:02.536832 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33977
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.126.keytab
--principal=oryx/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:02.839829 19152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:02.840094 19152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:02.840221 19152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:02.847273 19152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:59:02.847366 19152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:02.847436 19152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:02.847507 19152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:59:02.847561 19152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:59:02.857215 19152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.126.keytab
--principal=oryx/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:02.858492 19152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:02.859725 19152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:02.870864 19157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:02.870929 19160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:02.870880 19158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:02.871768 19152 server_base.cc:1034] running on GCE node
I20260928 02:59:02.872473 19152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:02.873795 19152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:02.874992 19152 hybrid_clock.cc:648] HybridClock initialized: now 1790564342874944 us; error 30 us; skew 500 ppm
I20260928 02:59:02.875306 19152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:02 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564342, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:02.880863 19152 init.cc:377] Logged in from keytab as oryx/127.12.182.126@KRBTEST.COM (short username oryx)
I20260928 02:59:02.882236 19152 webserver.cc:460] Webserver started at http://127.12.182.126:33105/ using document root <none> and password file <none>
I20260928 02:59:02.882841 19152 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:02.882933 19152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:02.883167 19152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:02.886027 19152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "1cba5416a93846beb848f485afb759a0"
format_stamp: "Formatted at 2026-09-28 02:59:02 on dist-test-slave-k5rg"
server_key: "fce1e613ceb59440673770120f014b0d"
server_key_iv: "24d3ac99f7c352df35738e001a75f781"
server_key_version: "encryptionkey@0"
I20260928 02:59:02.886686 19152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "1cba5416a93846beb848f485afb759a0"
format_stamp: "Formatted at 2026-09-28 02:59:02 on dist-test-slave-k5rg"
server_key: "fce1e613ceb59440673770120f014b0d"
server_key_iv: "24d3ac99f7c352df35738e001a75f781"
server_key_version: "encryptionkey@0"
I20260928 02:59:02.891336 19152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:59:02.894400 19167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:02.895663 19152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:59:02.895807 19152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "1cba5416a93846beb848f485afb759a0"
format_stamp: "Formatted at 2026-09-28 02:59:02 on dist-test-slave-k5rg"
server_key: "fce1e613ceb59440673770120f014b0d"
server_key_iv: "24d3ac99f7c352df35738e001a75f781"
server_key_version: "encryptionkey@0"
I20260928 02:59:02.895967 19152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:02.920964 19152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:02.924675 19152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:02.924918 19152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:02.941385 19152 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:33977
I20260928 02:59:02.941418 19218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:33977 every 8 connection(s)
I20260928 02:59:02.942652 19152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:59:02.946151 19219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:02.952437 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 19152
I20260928 02:59:02.952611 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:59:02.953302 13017 external_mini_cluster.cc:1428] Setting key d6cbcc39e49fbe6a4d1d5a38252b6127
I20260928 02:59:02.956028 19219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0: Bootstrap starting.
I20260928 02:59:02.959308 19219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:02.960433 19219 log.cc:824] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:02.963100 19219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0: No bootstrap required, opened a new log
Sep 28 02:59:02 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:02.971956 19219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } }
I20260928 02:59:02.972338 19219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:02.972450 19219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cba5416a93846beb848f485afb759a0, State: Initialized, Role: FOLLOWER
I20260928 02:59:02.973091 19219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } }
I20260928 02:59:02.973361 19219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:59:02.973503 19219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:59:02.973711 19219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:02.975418 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:02.955129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.12.182.126:33977)
0928 02:59:02.955842 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:02.955852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:02.955892 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:59:02.957571 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:02.957597 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:02.957989 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:02.958288 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:02.960003 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:02.961361 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:02.962609 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:02.963596 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:02.963773 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:02.967288 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:02.967299 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:02.967318 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:02.967365 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:02.971164 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:02.971750 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:02.971758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:02.971766 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:02.971965 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:02.972347 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:02.972352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:02.972355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:02.973016 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:02.973318 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:02.973652 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 02:59:02.974085 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":323,"threads_started":1}
I20260928 02:59:02.976848 19219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } }
I20260928 02:59:02.977322 19219 leader_election.cc:302] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cba5416a93846beb848f485afb759a0; no voters:
I20260928 02:59:02.977996 19219 leader_election.cc:288] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:59:02.978133 19224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:02.978835 19224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 1 LEADER]: Becoming Leader. State: Replica: 1cba5416a93846beb848f485afb759a0, State: Running, Role: LEADER
I20260928 02:59:02.979405 19224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } }
I20260928 02:59:02.980068 19219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:59:02.981958 19225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } }
I20260928 02:59:02.981935 19226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cba5416a93846beb848f485afb759a0. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } }
I20260928 02:59:02.982321 19225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:02.982435 19226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:02.984408 19229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:59:02.989827 19229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:59:02.998092 19229 catalog_manager.cc:1347] Generated new cluster ID: 2539384aab9c4d7db2df51240b308ee4
I20260928 02:59:02.998206 19229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:59:03.005582 19229 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:59:03.007035 19229 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:59:03.013305 19229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0: Generated new TSK 0
I20260928 02:59:03.014078 19229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab.
Entry for principal oryx/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab.
I20260928 02:59:03.068876 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33977
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab
--principal=oryx/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:03.371716 19245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:03.371973 19245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:03.372092 19245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:03.378949 19245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:03.379042 19245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:03.379181 19245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:59:03.388523 19245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab
--principal=oryx/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:03.389647 19245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:03.390841 19245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:03.402752 19250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:03.402705 19253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:03.402786 19251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:03.403386 19245 server_base.cc:1034] running on GCE node
I20260928 02:59:03.403949 19245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:03.404796 19245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:03.406002 19245 hybrid_clock.cc:648] HybridClock initialized: now 1790564343405957 us; error 36 us; skew 500 ppm
I20260928 02:59:03.406287 19245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:03 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564343, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:03.409204 19245 init.cc:377] Logged in from keytab as oryx/127.12.182.65@KRBTEST.COM (short username oryx)
I20260928 02:59:03.410512 19245 webserver.cc:460] Webserver started at http://127.12.182.65:45423/ using document root <none> and password file <none>
I20260928 02:59:03.411124 19245 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:03.411237 19245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:03.411509 19245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:03.414198 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "ad760c46b31745509ad8dc6f86e8f385"
format_stamp: "Formatted at 2026-09-28 02:59:03 on dist-test-slave-k5rg"
server_key: "5cc26b244a62b646d8feaeb381a1912a"
server_key_iv: "52d3b2828e5c7535fed6bfe965973dd5"
server_key_version: "encryptionkey@0"
I20260928 02:59:03.414907 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "ad760c46b31745509ad8dc6f86e8f385"
format_stamp: "Formatted at 2026-09-28 02:59:03 on dist-test-slave-k5rg"
server_key: "5cc26b244a62b646d8feaeb381a1912a"
server_key_iv: "52d3b2828e5c7535fed6bfe965973dd5"
server_key_version: "encryptionkey@0"
I20260928 02:59:03.419502 19245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:59:03.422570 19260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:03.423760 19245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 02:59:03.423913 19245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "ad760c46b31745509ad8dc6f86e8f385"
format_stamp: "Formatted at 2026-09-28 02:59:03 on dist-test-slave-k5rg"
server_key: "5cc26b244a62b646d8feaeb381a1912a"
server_key_iv: "52d3b2828e5c7535fed6bfe965973dd5"
server_key_version: "encryptionkey@0"
I20260928 02:59:03.424077 19245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:03.447408 19245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:03.451242 19245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:03.451567 19245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:03.452482 19245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:03.453805 19245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:03.453879 19245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:03.453987 19245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:03.454035 19245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:03.482919 19245 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:45413
I20260928 02:59:03.483021 19372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:45413 every 8 connection(s)
I20260928 02:59:03.484233 19245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:59:03.488155 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 19245
I20260928 02:59:03.488368 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:59:03.489163 13017 external_mini_cluster.cc:1428] Setting key 76e8410e60489c6cf2d48499ab8bbb00
Sep 28 02:59:03 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564343, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:03.504554 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:03.486598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:44543 (local address 127.12.182.126:33977)
0928 02:59:03.486991 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:03.486995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:03.487992 (+ 997us) server_negotiation.cc:407] Connection header received
0928 02:59:03.490263 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:03.490268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:03.490345 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:03.490445 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:03.492282 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:03.492986 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:03.494953 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:03.495370 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:03.495472 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:03.499250 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:03.499255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:03.499259 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:03.499290 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:03.501707 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:03.502501 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:03.502504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:03.502507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:03.502610 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:03.503005 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:03.503008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:03.503010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:03.503254 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:03.503353 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:03.504124 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 02:59:03.504291 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":193,"threads_started":1}
I20260928 02:59:03.505946 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:03.487128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.65:44543)
0928 02:59:03.487805 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:03.487859 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:03.490015 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:03.490694 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:03.490705 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:03.491376 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:03.492100 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:03.492124 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:03.493133 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:03.493138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:03.494801 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:03.494815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:03.495572 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:03.495578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:03.495824 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:03.496482 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:03.496507 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:03.499050 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:03.501839 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:03.501848 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:03.501877 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:03.502407 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:03.502689 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:03.502692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:03.502695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:03.502924 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:03.503365 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:03.503372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:03.503745 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 02:59:03.504054 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 02:59:03.504564 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":248,"threads_started":1}
I20260928 02:59:03.507462 19373 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33977
I20260928 02:59:03.507723 19373 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:03.508399 19373 heartbeater.cc:511] Master 127.12.182.126:33977 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.12.182.66@KRBTEST.COM; defaulting to no policy
I20260928 02:59:03.510386 19184 ts_manager.cc:194] Registered new tserver with Master: ad760c46b31745509ad8dc6f86e8f385 (127.12.182.65:45413)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.66@KRBTEST.COM" created.
I20260928 02:59:03.512012 19184 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:44543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab.
Entry for principal oryx/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab.
I20260928 02:59:03.532995 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33977
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab
--principal=oryx/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:03.833729 19379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:03.834059 19379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:03.834182 19379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:03.841085 19379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:03.841178 19379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:03.841290 19379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:59:03.850907 19379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab
--principal=oryx/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:03.851956 19379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:03.853123 19379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:03.865159 19385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:03.865269 19384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:03.865242 19387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:03.865952 19379 server_base.cc:1034] running on GCE node
I20260928 02:59:03.866490 19379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:03.867360 19379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:03.868551 19379 hybrid_clock.cc:648] HybridClock initialized: now 1790564343868510 us; error 36 us; skew 500 ppm
I20260928 02:59:03.868815 19379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:03 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564343, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:03.871778 19379 init.cc:377] Logged in from keytab as oryx/127.12.182.66@KRBTEST.COM (short username oryx)
I20260928 02:59:03.873056 19379 webserver.cc:460] Webserver started at http://127.12.182.66:41827/ using document root <none> and password file <none>
I20260928 02:59:03.873703 19379 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:03.873809 19379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:03.874082 19379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:03.876773 19379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "f29a5bb2b0074c3fbd25a6be50134e22"
format_stamp: "Formatted at 2026-09-28 02:59:03 on dist-test-slave-k5rg"
server_key: "8b55db6ae022dc38f01c8288d01de835"
server_key_iv: "666f2e139386d48dbb60243d8b62a255"
server_key_version: "encryptionkey@0"
I20260928 02:59:03.877497 19379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "f29a5bb2b0074c3fbd25a6be50134e22"
format_stamp: "Formatted at 2026-09-28 02:59:03 on dist-test-slave-k5rg"
server_key: "8b55db6ae022dc38f01c8288d01de835"
server_key_iv: "666f2e139386d48dbb60243d8b62a255"
server_key_version: "encryptionkey@0"
I20260928 02:59:03.882256 19379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:59:03.885483 19394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:03.886726 19379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:59:03.886898 19379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "f29a5bb2b0074c3fbd25a6be50134e22"
format_stamp: "Formatted at 2026-09-28 02:59:03 on dist-test-slave-k5rg"
server_key: "8b55db6ae022dc38f01c8288d01de835"
server_key_iv: "666f2e139386d48dbb60243d8b62a255"
server_key_version: "encryptionkey@0"
I20260928 02:59:03.887081 19379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:03.904549 19379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:03.908375 19379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:03.908619 19379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:03.909482 19379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:03.910742 19379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:03.910817 19379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:03.910941 19379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:03.910992 19379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:03.939340 19379 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:34761
I20260928 02:59:03.939375 19506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:34761 every 8 connection(s)
I20260928 02:59:03.940521 19379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:59:03.950053 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 19379
I20260928 02:59:03.950227 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:59:03.950690 13017 external_mini_cluster.cc:1428] Setting key a17ff140ca08f612da36a8a2fa37c21f
Sep 28 02:59:03 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564343, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:03.961211 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:03.943096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:43153 (local address 127.12.182.126:33977)
0928 02:59:03.943246 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:03.943250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:03.944503 (+ 1253us) server_negotiation.cc:407] Connection header received
0928 02:59:03.946259 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:03.946265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:03.946370 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:03.946456 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:03.947881 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:03.948610 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:03.950373 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:03.951036 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:03.951169 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:03.954636 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:03.954641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:03.954644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:03.954672 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:03.957750 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:03.958531 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:03.958537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:03.958556 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:03.958706 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:03.959169 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:03.959174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:03.959178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:03.959503 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:03.959663 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:03.960633 (+ 970us) server_negotiation.cc:299] Negotiation successful
0928 02:59:03.960876 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:59:03.963243 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:03.943623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.66:43153)
0928 02:59:03.944355 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:03.944395 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:03.946008 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:03.946685 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:03.946693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:03.947202 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:03.947719 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:03.947735 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:03.948767 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:03.948770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:03.950235 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:03.950250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:03.951284 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:03.951292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:03.951521 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:03.952379 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:03.952413 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:03.954440 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:03.957876 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:03.957885 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:03.957899 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:03.958403 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:03.958800 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:03.958805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:03.958809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:03.959043 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:03.959736 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:03.959743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:03.960162 (+ 419us) client_negotiation.cc:769] Sending connection context
0928 02:59:03.961591 (+ 1429us) client_negotiation.cc:240] Negotiation successful
0928 02:59:03.962119 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":316,"threads_started":1}
I20260928 02:59:03.964671 19507 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33977
I20260928 02:59:03.964989 19507 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:03.965718 19507 heartbeater.cc:511] Master 127.12.182.126:33977 requested a full tablet report, sending...
I20260928 02:59:03.967099 19184 ts_manager.cc:194] Registered new tserver with Master: f29a5bb2b0074c3fbd25a6be50134e22 (127.12.182.66:34761)
I20260928 02:59:03.968110 19184 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:43153
WARNING: no policy specified for oryx/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab.
Entry for principal oryx/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab.
I20260928 02:59:03.997020 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33977
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab
--principal=oryx/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:04.299165 19512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:04.299453 19512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:04.299588 19512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:04.306646 19512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:04.306754 19512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:04.306905 19512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:59:04.316056 19512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab
--principal=oryx/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:04.317284 19512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:04.318550 19512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:04.330538 19517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:04.330571 19520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:04.330597 19518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:04.331315 19512 server_base.cc:1034] running on GCE node
I20260928 02:59:04.331893 19512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:04.332737 19512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:04.333981 19512 hybrid_clock.cc:648] HybridClock initialized: now 1790564344333925 us; error 46 us; skew 500 ppm
I20260928 02:59:04.334285 19512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:04 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564344, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:04.337174 19512 init.cc:377] Logged in from keytab as oryx/127.12.182.67@KRBTEST.COM (short username oryx)
I20260928 02:59:04.338579 19512 webserver.cc:460] Webserver started at http://127.12.182.67:35141/ using document root <none> and password file <none>
I20260928 02:59:04.339228 19512 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:04.339344 19512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:04.339617 19512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:04.342468 19512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "86a5462cbbc34ec389f82d89abfe96c9"
format_stamp: "Formatted at 2026-09-28 02:59:04 on dist-test-slave-k5rg"
server_key: "57f88813160772d62e5fef7c866dbbcc"
server_key_iv: "99d5360204f6af275865a0396ee6c5f2"
server_key_version: "encryptionkey@0"
I20260928 02:59:04.343195 19512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "86a5462cbbc34ec389f82d89abfe96c9"
format_stamp: "Formatted at 2026-09-28 02:59:04 on dist-test-slave-k5rg"
server_key: "57f88813160772d62e5fef7c866dbbcc"
server_key_iv: "99d5360204f6af275865a0396ee6c5f2"
server_key_version: "encryptionkey@0"
I20260928 02:59:04.347945 19512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 02:59:04.351322 19527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:04.352566 19512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:59:04.352712 19512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "86a5462cbbc34ec389f82d89abfe96c9"
format_stamp: "Formatted at 2026-09-28 02:59:04 on dist-test-slave-k5rg"
server_key: "57f88813160772d62e5fef7c866dbbcc"
server_key_iv: "99d5360204f6af275865a0396ee6c5f2"
server_key_version: "encryptionkey@0"
I20260928 02:59:04.352993 19512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:04.371019 19512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:04.374753 19512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:04.375006 19512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:04.375849 19512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:04.377089 19512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:04.377163 19512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:04.377285 19512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:04.377336 19512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:04.406693 19512 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:42661
I20260928 02:59:04.406724 19639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:42661 every 8 connection(s)
I20260928 02:59:04.407866 19512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:59:04.413087 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 19512
I20260928 02:59:04.413269 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:59:04.413780 13017 external_mini_cluster.cc:1428] Setting key 7dd2a2393c2d58fc0475c556ac4791e6
Sep 28 02:59:04 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564344, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:04.426645 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.410208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:53149 (local address 127.12.182.126:33977)
0928 02:59:04.410371 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:04.410375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:04.411485 (+ 1110us) server_negotiation.cc:407] Connection header received
0928 02:59:04.413152 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:04.413158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:04.413246 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:04.413358 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:04.414901 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.415902 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.417690 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.418126 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.418217 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.421315 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:04.421320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:04.421323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:04.421356 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:04.423913 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.424589 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.424593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.424596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.424724 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.425153 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.425156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.425158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.425432 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:04.425545 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:04.426229 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 02:59:04.426410 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:59:04.427758 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.410695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.67:53149)
0928 02:59:04.411341 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:04.411375 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:04.412890 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:04.413567 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:04.413574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:04.414108 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:04.414725 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.414742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.416054 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.416060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.417537 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.417550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.418295 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.418300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.418509 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.419107 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:04.419133 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:04.421113 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:04.424047 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:04.424053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:04.424066 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:04.424493 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:04.424848 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:04.424852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:04.424855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:04.425049 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:04.425542 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:04.425549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:04.425888 (+ 339us) client_negotiation.cc:769] Sending connection context
0928 02:59:04.426121 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 02:59:04.426531 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":249,"threads_started":1}
I20260928 02:59:04.429219 19640 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33977
I20260928 02:59:04.429512 19640 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:04.430254 19640 heartbeater.cc:511] Master 127.12.182.126:33977 requested a full tablet report, sending...
I20260928 02:59:04.431564 19183 ts_manager.cc:194] Registered new tserver with Master: 86a5462cbbc34ec389f82d89abfe96c9 (127.12.182.67:42661)
I20260928 02:59:04.432384 19183 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:53149
I20260928 02:59:04.441027 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.125.keytab.
Entry for principal oryx/127.12.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.125.keytab.
I20260928 02:59:04.481925 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.125:37419
--webserver_interface=127.12.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.125.keytab
--principal=oryx/127.12.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33977,127.12.182.125:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:59:04.515136 19373 heartbeater.cc:503] Master 127.12.182.126:33977 was elected leader, sending a full tablet report...
W20260928 02:59:04.786077 19645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:04.786335 19645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:04.786427 19645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:04.795661 19645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:59:04.795761 19645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:04.795810 19645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:04.795862 19645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:59:04.795903 19645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:59:04.805922 19645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33977,127.12.182.125:37419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.125.keytab
--principal=oryx/127.12.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.125:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
W20260928 02:59:04.806223 19645 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.182.126:33977,127.12.182.125:37419'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:59:04.807147 19645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:04.808290 19645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:04.818849 19653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:04.818890 19651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:04.818925 19650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:04.819643 19645 server_base.cc:1034] running on GCE node
I20260928 02:59:04.820307 19645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:04.821708 19645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:04.822923 19645 hybrid_clock.cc:648] HybridClock initialized: now 1790564344822855 us; error 49 us; skew 500 ppm
I20260928 02:59:04.823226 19645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:04 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564344, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:04.826151 19645 init.cc:377] Logged in from keytab as oryx/127.12.182.125@KRBTEST.COM (short username oryx)
I20260928 02:59:04.827524 19645 webserver.cc:460] Webserver started at http://127.12.182.125:43965/ using document root <none> and password file <none>
I20260928 02:59:04.828188 19645 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:04.828289 19645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:04.828562 19645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:04.831331 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data/instance:
uuid: "fbe76f76ea1b4f9b97ccab85094b50d7"
format_stamp: "Formatted at 2026-09-28 02:59:04 on dist-test-slave-k5rg"
server_key: "5c070b8281c3fcb16b2649d0c98934ba"
server_key_iv: "8fbf503c7b8d1940842b4067324ab02a"
server_key_version: "encryptionkey@0"
I20260928 02:59:04.832050 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal/instance:
uuid: "fbe76f76ea1b4f9b97ccab85094b50d7"
format_stamp: "Formatted at 2026-09-28 02:59:04 on dist-test-slave-k5rg"
server_key: "5c070b8281c3fcb16b2649d0c98934ba"
server_key_iv: "8fbf503c7b8d1940842b4067324ab02a"
server_key_version: "encryptionkey@0"
I20260928 02:59:04.836794 19645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:59:04.840000 19660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:04.841217 19645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:59:04.841365 19645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal
uuid: "fbe76f76ea1b4f9b97ccab85094b50d7"
format_stamp: "Formatted at 2026-09-28 02:59:04 on dist-test-slave-k5rg"
server_key: "5c070b8281c3fcb16b2649d0c98934ba"
server_key_iv: "8fbf503c7b8d1940842b4067324ab02a"
server_key_version: "encryptionkey@0"
I20260928 02:59:04.841548 19645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:04.867678 19645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:04.871871 19645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:04.872109 19645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:04.888749 19645 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.125:37419
I20260928 02:59:04.888757 19711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.125:37419 every 8 connection(s)
I20260928 02:59:04.890089 19645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
I20260928 02:59:04.893477 19712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:04.899403 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 19645
I20260928 02:59:04.899488 19712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:59:04 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564344, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.125@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
Sep 28 02:59:04 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.125@KRBTEST.COM
I20260928 02:59:04.917145 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.901983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50302 (local address 127.12.182.126:33977)
0928 02:59:04.902137 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:04.902141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:04.902828 (+ 687us) server_negotiation.cc:407] Connection header received
0928 02:59:04.904172 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:04.904178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:04.904260 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:04.904347 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:04.905591 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.906517 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.907959 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.908342 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.908447 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.911748 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:04.911752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:04.911762 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:04.911794 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:04.914372 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.915084 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.915089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.915093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.915211 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.915574 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.915579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.915583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.915882 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:04.916005 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:04.916616 (+ 611us) server_negotiation.cc:299] Negotiation successful
0928 02:59:04.916820 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:59:04.918397 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.901293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.12.182.125:37419)
0928 02:59:04.902058 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:04.902070 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:04.902117 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:59:04.903895 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:04.903921 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:04.904153 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:04.904320 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:04.905702 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.906795 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.907959 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.908442 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.908603 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.912700 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:04.912707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:04.912717 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:04.912744 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:04.915288 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.915916 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.915922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.915924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.916041 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.916384 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.916389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.916391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.916706 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:04.916969 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:04.917285 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:59:04.917511 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"spinlock_wait_cycles":22272,"thread_start_us":309,"threads_started":1}
I20260928 02:59:04.918403 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.902048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50302)
0928 02:59:04.902694 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:04.902729 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:04.903921 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:04.904455 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:04.904462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:04.905054 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:04.905433 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.905451 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.906653 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.906657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.907823 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.907835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.908549 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.908556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.908685 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.909292 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:04.909320 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:04.911570 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:04.914480 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:04.914491 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:04.914501 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:04.914986 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:04.915305 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:04.915310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:04.915314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:04.915478 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:04.916005 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:04.916012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:04.916286 (+ 274us) client_negotiation.cc:769] Sending connection context
0928 02:59:04.916530 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 02:59:04.916984 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":348,"threads_started":1}
I20260928 02:59:04.920127 19712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:59:04 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564344, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.125@KRBTEST.COM for oryx/127.12.182.125@KRBTEST.COM
I20260928 02:59:04.931185 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.920883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:37419 (local address 127.0.0.1:44128)
0928 02:59:04.921046 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:04.921063 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:04.921130 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:04.921389 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:04.921392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:04.921721 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:04.921850 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.921858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.922730 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.922733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.923579 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.923589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.924218 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.924222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.924339 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.924828 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:04.924843 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:04.926661 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:04.929202 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:04.929205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:04.929208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:04.929589 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:04.929928 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:04.929931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:04.929934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:04.930054 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:04.930538 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:04.930542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:04.930662 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 02:59:04.930767 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:59:04.930956 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:59:04.931330 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.920917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.12.182.125:37419)
0928 02:59:04.921075 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:04.921079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:04.921095 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:59:04.921203 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:04.921206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:04.921253 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:04.921321 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:04.921963 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.922616 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.923699 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.924102 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.924194 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.926839 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:04.926843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:04.926846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:04.926875 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:04.929076 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.929709 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.929713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.929715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.929823 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.930157 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.930160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.930162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.930425 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:04.930527 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:04.930855 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:59:04.931064 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":59}
I20260928 02:59:04.935614 19712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: Bootstrap starting.
I20260928 02:59:04.938503 19712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:04.939702 19712 log.cc:824] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:04.942548 19712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: No bootstrap required, opened a new log
I20260928 02:59:04.949831 19712 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } }
I20260928 02:59:04.950103 19712 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:04.950196 19712 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe76f76ea1b4f9b97ccab85094b50d7, State: Initialized, Role: FOLLOWER
I20260928 02:59:04.950716 19712 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } }
I20260928 02:59:04.951469 19718 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } } }
I20260928 02:59:04.951745 19718 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:59:04.952278 19712 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:59:04.959800 19729 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:59:04.959909 19729 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:59:04.962249 19645 master_runner.cc:428] Detected that this master fbe76f76ea1b4f9b97ccab85094b50d7 is joining an existing cluster
I20260928 02:59:04.962347 19645 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.182.125:37419
I20260928 02:59:04.970568 19507 heartbeater.cc:503] Master 127.12.182.126:33977 was elected leader, sending a full tablet report...
I20260928 02:59:04.981546 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.966081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.12.182.125:37419)
0928 02:59:04.966441 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:04.966447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:04.967335 (+ 888us) server_negotiation.cc:407] Connection header received
0928 02:59:04.967480 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:04.967485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:04.967544 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:04.967683 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:04.968900 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.969986 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.971775 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.972421 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.972514 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.975142 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:04.975148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:04.975152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:04.975189 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:04.978848 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.979585 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.979589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.979592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.979711 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.980076 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.980080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.980083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.980396 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:04.980520 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:04.980989 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 02:59:04.981195 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260928 02:59:04.981853 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.965798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50310 (local address 127.12.182.126:33977)
0928 02:59:04.966281 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:04.966289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:04.966974 (+ 685us) server_negotiation.cc:407] Connection header received
0928 02:59:04.967274 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:04.967280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:04.967363 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:04.967476 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:04.968569 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.969640 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.971699 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.972344 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.972455 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.974979 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:04.974985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:04.974990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:04.975026 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:04.978333 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.979484 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.979490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.979494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.979641 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:04.980086 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:04.980091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:04.980094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:04.980470 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:04.980594 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:04.981246 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 02:59:04.981491 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260928 02:59:04.982604 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.966021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:37419 (local address 127.0.0.1:44132)
0928 02:59:04.967254 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:04.967268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:04.967375 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:04.967736 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:04.967745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:04.968438 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:04.968748 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.968765 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.970160 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.970163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.971643 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.971656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.972808 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.972814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.972993 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.973901 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:04.973923 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:04.974984 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:04.978974 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:04.978979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:04.978983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:04.979474 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:04.979817 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:04.979820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:04.979823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:04.979981 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:04.980593 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:04.980596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:04.980732 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:59:04.981793 (+ 1061us) client_negotiation.cc:240] Negotiation successful
0928 02:59:04.982177 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"mutex_wait_us":64,"thread_start_us":943,"threads_started":1}
I20260928 02:59:04.982887 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.965894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50310)
0928 02:59:04.966865 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:04.966887 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:04.967087 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:04.967511 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:04.967516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:04.968098 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:04.968436 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.968446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.969766 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.969771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.971533 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.971547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.972727 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.972733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.972922 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.973566 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:04.973589 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:04.974773 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:04.978495 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:04.978500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:04.978504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:04.979338 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:04.979783 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:04.979787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:04.979791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:04.979959 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:04.980716 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:04.980720 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:04.980911 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 02:59:04.982027 (+ 1116us) client_negotiation.cc:240] Negotiation successful
0928 02:59:04.982368 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":67,"spinlock_wait_cycles":31104,"thread_start_us":686,"threads_started":1}
WARNING: no policy specified for oryx/127.12.182.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.124@KRBTEST.COM" created.
Sep 28 02:59:04 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564342, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.126@KRBTEST.COM for oryx/127.12.182.125@KRBTEST.COM
I20260928 02:59:05.006426 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.989029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.12.182.125:37419)
0928 02:59:04.989172 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:04.989176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:04.989806 (+ 630us) server_negotiation.cc:407] Connection header received
0928 02:59:04.990120 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:04.990126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:04.990225 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:04.990316 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:04.991793 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.992869 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.994855 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.995446 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.995541 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.999606 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:04.999612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:04.999617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:04.999654 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.003293 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.004160 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.004165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.004169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.004327 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.004720 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.004725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.004729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.005071 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.005232 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.005818 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.006056 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:59:05.006428 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:04.989039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:37419 (local address 127.0.0.1:44146)
0928 02:59:04.989672 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:04.989727 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:04.989920 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:04.990540 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:04.990548 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:04.991347 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:04.991616 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.991629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.993083 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.993088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.994683 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:04.994697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:04.995611 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:04.995617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:04.995846 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:04.996424 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:04.996454 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:04.999396 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:05.003441 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.003453 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.003460 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.003999 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.004436 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.004440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.004443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.004595 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.005282 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:05.005290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:05.005490 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.005682 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.006015 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":249,"threads_started":1}
I20260928 02:59:05.008706 19183 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.182.125:37419: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7"
member_type: NON_VOTER
last_known_addr {
host: "127.12.182.125"
port: 37419
}
attrs {
promote: true
}
}
dest_uuid: "1cba5416a93846beb848f485afb759a0"
cas_config_opid_index: -1
I20260928 02:59:05.009841 19183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } }
I20260928 02:59:05.016873 19741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fbe76f76ea1b4f9b97ccab85094b50d7 (127.12.182.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } } }
I20260928 02:59:05.018162 19687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:05.020931 19687 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 FOLLOWER]: Refusing update from remote peer 1cba5416a93846beb848f485afb759a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:59:05.021379 19741 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.182.125:37419
I20260928 02:59:05.021977 19741 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:05.021903 19742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } } }
I20260928 02:59:05.022501 19742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:05.022862 19645 master.cc:537] Master@127.12.182.125:37419 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.124.keytab.
Entry for principal oryx/127.12.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.124.keytab.
I20260928 02:59:05.025236 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.124:40133
--webserver_interface=127.12.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.124.keytab
--principal=oryx/127.12.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33977,127.12.182.125:37419,127.12.182.124:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:05.028331 19169 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 02:59:05.029906 19169 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 -> Peer fbe76f76ea1b4f9b97ccab85094b50d7 (127.12.182.125:37419): Couldn't send request to peer fbe76f76ea1b4f9b97ccab85094b50d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:59:05.034690 19645 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:05.035197 19645 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:05.035465 19645 tablet_replica.cc:330] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: stopping tablet replica
I20260928 02:59:05.042502 19645 master.cc:552] Master@127.12.182.125:37419 shutdown complete.
I20260928 02:59:05.042585 19645 master_runner.cc:305] Clearing existing system tablet
I20260928 02:59:05.042985 19645 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:05.043075 19645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:05.045950 19645 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 02:59:05.047544 19748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:05.047909 19645 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:05.048183 19645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 02:59:05.048261 19645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal
uuid: "fbe76f76ea1b4f9b97ccab85094b50d7"
format_stamp: "Formatted at 2026-09-28 02:59:04 on dist-test-slave-k5rg"
server_key: "5c070b8281c3fcb16b2649d0c98934ba"
server_key_iv: "8fbf503c7b8d1940842b4067324ab02a"
server_key_version: "encryptionkey@0"
I20260928 02:59:05.053069 19645 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:05.059759 19645 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:59:05.060055 19645 log.cc:1196] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:59:05.060472 19645 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: Deleting consensus metadata
I20260928 02:59:05.060729 19645 master_runner.cc:315] Copying system tablet from 127.12.182.126:33977
I20260928 02:59:05.070864 19645 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: tablet copy: Beginning tablet copy session from remote peer at address 127.12.182.126:33977
I20260928 02:59:05.086683 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.071727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.12.182.126:33977)
0928 02:59:05.071929 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.071935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.072422 (+ 487us) server_negotiation.cc:407] Connection header received
0928 02:59:05.072619 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.072625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.072722 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.072834 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.073906 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.074996 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.076737 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.077369 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.077497 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.079702 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.079709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.079714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.079807 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.083512 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.084347 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.084362 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.084366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.084530 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.084988 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.084993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.084996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.085410 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.085554 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.086000 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.086345 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:59:05.086695 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.071682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50314)
0928 02:59:05.072294 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:05.072312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:05.072464 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:05.072876 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:05.072896 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:05.073408 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:05.073715 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.073729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.075134 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.075140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.076583 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.076599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.077528 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.077535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.077743 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.078415 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:05.078438 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:05.079483 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:05.083659 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.083664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.083669 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.084192 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.084647 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.084653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.084657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.084840 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.085531 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:05.085536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:05.085744 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.085910 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.086243 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":363,"threads_started":1}
I20260928 02:59:05.087821 19204 tablet_copy_service.cc:140] P 1cba5416a93846beb848f485afb759a0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fbe76f76ea1b4f9b97ccab85094b50d7 ({username='oryx', principal='oryx/127.12.182.125@KRBTEST.COM'} at 127.0.0.1:50314)
I20260928 02:59:05.088152 19204 tablet_copy_service.cc:161] P 1cba5416a93846beb848f485afb759a0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fbe76f76ea1b4f9b97ccab85094b50d7 at {username='oryx', principal='oryx/127.12.182.125@KRBTEST.COM'} at 127.0.0.1:50314: session id = fbe76f76ea1b4f9b97ccab85094b50d7-00000000000000000000000000000000
I20260928 02:59:05.092641 19204 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:59:05.094930 19645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:59:05.101966 19645 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: tablet copy: Starting download of 0 data blocks...
I20260928 02:59:05.102280 19645 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: tablet copy: Starting download of 1 WAL segments...
I20260928 02:59:05.105036 19645 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:59:05.109266 19204 tablet_copy_service.cc:342] P 1cba5416a93846beb848f485afb759a0: Request end of tablet copy session fbe76f76ea1b4f9b97ccab85094b50d7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.182.125@KRBTEST.COM'} at 127.0.0.1:50314
I20260928 02:59:05.109499 19204 tablet_copy_service.cc:433] P 1cba5416a93846beb848f485afb759a0: ending tablet copy session fbe76f76ea1b4f9b97ccab85094b50d7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fbe76f76ea1b4f9b97ccab85094b50d7
I20260928 02:59:05.122476 19645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:05.125159 19755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:05.125739 19756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:05.133145 19758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:05.133490 19645 server_base.cc:1034] running on GCE node
I20260928 02:59:05.133868 19645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:05.134469 19645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:05.135694 19645 hybrid_clock.cc:648] HybridClock initialized: now 1790564345135609 us; error 48 us; skew 500 ppm
I20260928 02:59:05.135926 19645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:05 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564345, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:05.138056 19645 init.cc:377] Logged in from keytab as oryx/127.12.182.125@KRBTEST.COM (short username oryx)
I20260928 02:59:05.142474 19645 webserver.cc:460] Webserver started at http://127.12.182.125:36627/ using document root <none> and password file <none>
I20260928 02:59:05.142855 19645 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:05.142935 19645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:05.146099 19645 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:59:05.147996 19765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:05.148420 19645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:59:05.148561 19645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal
uuid: "fbe76f76ea1b4f9b97ccab85094b50d7"
format_stamp: "Formatted at 2026-09-28 02:59:04 on dist-test-slave-k5rg"
server_key: "5c070b8281c3fcb16b2649d0c98934ba"
server_key_iv: "8fbf503c7b8d1940842b4067324ab02a"
server_key_version: "encryptionkey@0"
I20260928 02:59:05.148716 19645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:05.192054 19645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:05.196015 19645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:05.224078 19645 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.125:37419
I20260928 02:59:05.224108 19816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.125:37419 every 8 connection(s)
I20260928 02:59:05.225189 19645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
I20260928 02:59:05.230425 19817 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:59:05.234534 19817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: Bootstrap starting.
I20260928 02:59:05.250510 19817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:59:05.251066 19817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: Bootstrap complete.
I20260928 02:59:05.252117 19817 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } }
I20260928 02:59:05.252393 19817 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fbe76f76ea1b4f9b97ccab85094b50d7, State: Initialized, Role: LEARNER
I20260928 02:59:05.252653 19817 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } }
I20260928 02:59:05.253443 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } } }
I20260928 02:59:05.253731 19817 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:59:05.253854 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:59:05.269091 19831 catalog_manager.cc:1259] Loaded cluster ID: 2539384aab9c4d7db2df51240b308ee4
I20260928 02:59:05.269301 19831 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: loading cluster ID for follower catalog manager: success
I20260928 02:59:05.272854 19831 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: acquiring CA information for follower catalog manager: success
I20260928 02:59:05.275252 19831 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:59:05.393626 19746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:05.393894 19746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:05.394014 19746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:05.400906 19746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:59:05.401002 19746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:05.401073 19746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:05.401150 19746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:59:05.401211 19746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:59:05.410667 19746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33977,127.12.182.125:37419,127.12.182.124:40133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.124.keytab
--principal=oryx/127.12.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.124:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:05.411753 19746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:05.412863 19746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:05.423410 19836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:05.423482 19837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:05.423525 19839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:05.424158 19746 server_base.cc:1034] running on GCE node
I20260928 02:59:05.424805 19746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:05.426066 19746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:05.427273 19746 hybrid_clock.cc:648] HybridClock initialized: now 1790564345427224 us; error 38 us; skew 500 ppm
I20260928 02:59:05.427573 19746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:05 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564345, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:05.430348 19746 init.cc:377] Logged in from keytab as oryx/127.12.182.124@KRBTEST.COM (short username oryx)
I20260928 02:59:05.431712 19746 webserver.cc:460] Webserver started at http://127.12.182.124:43979/ using document root <none> and password file <none>
I20260928 02:59:05.432312 19746 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:05.432420 19746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:05.432689 19746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:05.434684 19640 heartbeater.cc:503] Master 127.12.182.126:33977 was elected leader, sending a full tablet report...
I20260928 02:59:05.435575 19746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data/instance:
uuid: "b2b6b0674db54a908303ea35bce9f8a2"
format_stamp: "Formatted at 2026-09-28 02:59:05 on dist-test-slave-k5rg"
server_key: "fdc42b28597d9af6cffb51d72fb9dea2"
server_key_iv: "0f1c0f798d83c223a1e42dd9d05819ed"
server_key_version: "encryptionkey@0"
I20260928 02:59:05.436474 19746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal/instance:
uuid: "b2b6b0674db54a908303ea35bce9f8a2"
format_stamp: "Formatted at 2026-09-28 02:59:05 on dist-test-slave-k5rg"
server_key: "fdc42b28597d9af6cffb51d72fb9dea2"
server_key_iv: "0f1c0f798d83c223a1e42dd9d05819ed"
server_key_version: "encryptionkey@0"
I20260928 02:59:05.441048 19746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:59:05.444145 19846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:05.445359 19746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:59:05.445516 19746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal
uuid: "b2b6b0674db54a908303ea35bce9f8a2"
format_stamp: "Formatted at 2026-09-28 02:59:05 on dist-test-slave-k5rg"
server_key: "fdc42b28597d9af6cffb51d72fb9dea2"
server_key_iv: "0f1c0f798d83c223a1e42dd9d05819ed"
server_key_version: "encryptionkey@0"
I20260928 02:59:05.445731 19746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:05.463035 19746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:05.466811 19746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:05.467049 19746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:05.483419 19746 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.124:40133
I20260928 02:59:05.483491 19897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.124:40133 every 8 connection(s)
I20260928 02:59:05.484752 19746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
I20260928 02:59:05.488188 19898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:05.492624 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 19746
I20260928 02:59:05.494611 19898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:59:05 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564345, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.124@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
Sep 28 02:59:05 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.124@KRBTEST.COM
I20260928 02:59:05.511668 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.497645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.12.182.126:33977)
0928 02:59:05.497830 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.497835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.498060 (+ 225us) server_negotiation.cc:407] Connection header received
0928 02:59:05.498260 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.498265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.498338 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.498441 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.499696 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.500525 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.502084 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.502583 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.502672 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.506118 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.506123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.506126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.506155 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.508702 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.509520 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.509523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.509525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.509674 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.510106 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.510110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.510113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.510453 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.510597 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.511281 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.511440 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:59:05.512900 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.494549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.12.182.124:40133)
0928 02:59:05.495315 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.495329 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.495384 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:59:05.497493 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.497525 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.498012 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.498295 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.499637 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.501028 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.502297 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.502860 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.503012 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.506867 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.506874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.506884 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.506914 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.509943 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.510676 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.510683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.510686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.510831 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.511178 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.511183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.511186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.511563 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.511831 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.512159 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.512372 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":529,"thread_start_us":326,"threads_started":1}
I20260928 02:59:05.512876 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.497301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50316)
0928 02:59:05.497921 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:05.497960 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:05.498122 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:05.498530 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:05.498537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:05.499183 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:05.499541 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.499558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.500644 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.500649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.501953 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.501965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.502785 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.502792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.503020 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.503777 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:05.503797 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:05.505944 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:05.508820 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.508829 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.508839 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.509421 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.509774 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.509778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.509782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.509997 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.510573 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:05.510579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:05.510942 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.511188 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.511637 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":32,"thread_start_us":245,"threads_started":1}
I20260928 02:59:05.514984 19898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:59:05 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564345, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.124@KRBTEST.COM for oryx/127.12.182.125@KRBTEST.COM
I20260928 02:59:05.526997 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.515611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:37419 (local address 127.0.0.1:44154)
0928 02:59:05.515765 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:05.515781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:05.515877 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:05.516372 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:05.516376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:05.516672 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:05.516782 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.516790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.518236 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.518241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.519111 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.519122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.519785 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.519791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.519978 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.520408 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:05.520426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:05.522285 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:05.524803 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.524808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.524812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.525155 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.525508 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.525513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.525516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.525670 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.526123 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:05.526128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:05.526261 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.526370 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.526666 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:59:05.527012 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.515676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.12.182.125:37419)
0928 02:59:05.516134 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.516138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.516151 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:59:05.516195 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.516198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.516250 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.516336 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.516911 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.518115 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.519233 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.519659 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.519776 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.522468 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.522472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.522476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.522507 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.524677 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.525261 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.525266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.525268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.525408 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.525771 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.525774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.525776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.526017 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.526152 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.526478 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.526666 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":223,"threads_started":1}
I20260928 02:59:05.528292 19898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:59:05 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564345, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.124@KRBTEST.COM for oryx/127.12.182.124@KRBTEST.COM
I20260928 02:59:05.539482 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.528975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:40133 (local address 127.0.0.1:35368)
0928 02:59:05.529120 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:05.529137 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:05.529214 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:05.529476 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:05.529479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:05.529810 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:05.529938 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.529946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.530800 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.530803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.531609 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.531619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.532240 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.532244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.532394 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.532813 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:05.532827 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:05.534614 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:05.537372 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.537375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.537378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.537727 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.538034 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.538038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.538042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.538171 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.538611 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:05.538615 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:05.538730 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.538842 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.539085 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:59:05.539542 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.528980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.12.182.124:40133)
0928 02:59:05.529109 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.529113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.529174 (+ 61us) server_negotiation.cc:407] Connection header received
0928 02:59:05.529286 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.529290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.529339 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.529428 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.530059 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.530698 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.531717 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.532125 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.532216 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.534769 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.534773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.534784 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.534809 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.537241 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.537828 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.537831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.537834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.537942 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.538270 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.538273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.538275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.538513 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.538629 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.538943 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.539257 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":46}
I20260928 02:59:05.541276 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.536162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.12.182.125:37419)
0928 02:59:05.536424 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.536429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.536507 (+ 78us) server_negotiation.cc:407] Connection header received
0928 02:59:05.536671 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.536676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.536768 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.536856 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:05.537238 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.538038 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.539479 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.540527 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.540683 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.540847 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.541034 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.541124 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260928 02:59:05.541446 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.535892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:37419 (local address 127.0.0.1:44162)
0928 02:59:05.536411 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:05.536438 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:05.536560 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:05.536916 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:05.536921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:05.536948 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:05.537119 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.537128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.538175 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.538181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.539337 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.539350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.540687 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.540692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.540861 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.540918 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.541023 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.541155 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":296,"threads_started":1}
I20260928 02:59:05.542498 19792 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 02:59:05.543022 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cba5416a93846beb848f485afb759a0. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } } }
I20260928 02:59:05.543257 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:59:05.544497 19898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Bootstrap starting.
I20260928 02:59:05.547410 19898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:05.548336 19898 log.cc:824] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:05.550954 19898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: No bootstrap required, opened a new log
I20260928 02:59:05.557786 19898 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } }
I20260928 02:59:05.558073 19898 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:05.558174 19898 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b6b0674db54a908303ea35bce9f8a2, State: Initialized, Role: FOLLOWER
I20260928 02:59:05.558642 19898 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } }
I20260928 02:59:05.559379 19906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } } }
I20260928 02:59:05.559700 19906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:59:05.560353 19898 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:59:05.567312 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 19245
W20260928 02:59:05.572106 19917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:59:05.572326 19917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:59:05.572712 19746 master_runner.cc:428] Detected that this master b2b6b0674db54a908303ea35bce9f8a2 is joining an existing cluster
I20260928 02:59:05.572827 19746 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.182.124:40133
Sep 28 02:59:05 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.65@KRBTEST.COM
I20260928 02:59:05.593184 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.568291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.12.182.65:45413)
0928 02:59:05.568860 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.568866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.568888 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:59:05.568939 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.568944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.569125 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.569255 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.570919 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.572373 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.574296 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.575141 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.575271 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.584532 (+ 9261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.584549 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.584568 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.584628 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.588492 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.589333 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.589342 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.589345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.589519 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.590642 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.590647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.590650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.591080 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.591369 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.592504 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.592780 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":279,"threads_started":1}
I20260928 02:59:05.594767 19368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:05.611814 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.580571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.12.182.124:40133)
0928 02:59:05.580725 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.580731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.596840 (+ 16109us) server_negotiation.cc:407] Connection header received
0928 02:59:05.596966 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.596971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.597049 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.597154 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.598861 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.600092 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.602003 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.602702 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.602787 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.605156 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.605162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.605167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.605199 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.608773 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.609525 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.609530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.609534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.609717 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.610203 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.610208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.610210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.610563 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.610698 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.611176 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.611400 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:59:05.613080 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.580116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:40133 (local address 127.0.0.1:35376)
0928 02:59:05.596725 (+ 16609us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:05.596746 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:05.596866 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:05.597291 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:05.597297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:05.597973 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:05.598345 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.598365 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.600347 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.600352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.601854 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.601869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.603066 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.603072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.603226 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.603811 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:05.603861 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:05.604993 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:05.608933 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.608938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.608943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.609412 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.609835 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.609840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.609844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.609967 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.610775 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:05.610780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:05.610914 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.612311 (+ 1397us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.612627 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16492,"thread_start_us":16831,"threads_started":1}
I20260928 02:59:05.617314 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.576621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50320)
0928 02:59:05.590887 (+ 14266us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:05.590908 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:05.591111 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:05.591693 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:05.591701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:05.592409 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:05.593928 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.593940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.595274 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.595280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.596911 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.596922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.598074 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.598080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.598253 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.598953 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:05.598975 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:05.600151 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:05.614199 (+ 14048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.614204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.614207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.614652 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.615110 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.615115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.615118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.615334 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.616239 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:05.616244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:05.616449 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.616608 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.616894 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14113,"mutex_wait_us":1192,"thread_start_us":13933,"threads_started":1}
I20260928 02:59:05.617338 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.576944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.12.182.126:33977)
0928 02:59:05.577234 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.577239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.591001 (+ 13762us) server_negotiation.cc:407] Connection header received
0928 02:59:05.591268 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.591275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.591359 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.591425 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.594092 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.595117 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.597051 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.597709 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.597803 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.600351 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.600358 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.600363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.600400 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.614031 (+ 13631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.614799 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.614804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.614808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.614967 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.615539 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.615545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.615548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.615978 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.616149 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.616742 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.616982 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260928 02:59:05.625444 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.577435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.12.182.125:37419)
0928 02:59:05.577598 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.577622 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.587396 (+ 9774us) server_negotiation.cc:407] Connection header received
0928 02:59:05.587931 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.587936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.588018 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.588088 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.589518 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.590644 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.592618 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.593242 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.593339 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.619166 (+ 25827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.619174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.619179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.619222 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.622488 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.623208 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.623213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.623217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.623363 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.623786 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.623791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.623794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.624154 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.624266 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.624847 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.625104 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:59:05.626410 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.576585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:37419 (local address 127.0.0.1:44172)
0928 02:59:05.587290 (+ 10705us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:05.587312 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:05.587748 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:05.588294 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:05.588299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:05.588918 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:05.589283 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.589296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.590894 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.590899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.592462 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.592476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.600564 (+ 8088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.600570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.600746 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.617873 (+ 17127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:05.617902 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:05.618940 (+ 1038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:05.622642 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.622647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.622651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.623078 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.623501 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.623504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.623507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.623671 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.624356 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:05.624360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:05.624568 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.625697 (+ 1129us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.625994 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10569,"mutex_wait_us":3,"thread_start_us":10822,"threads_started":1}
Sep 28 02:59:05 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564342, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.126@KRBTEST.COM for oryx/127.12.182.124@KRBTEST.COM
I20260928 02:59:05.650398 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.634796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:40133 (local address 127.0.0.1:35392)
0928 02:59:05.635020 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:05.635040 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:05.635153 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:05.635757 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:05.635762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:05.636210 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:05.636372 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.636419 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.637941 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.637946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.639357 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.639373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.640504 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.640510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.640683 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.641361 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:05.641383 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:05.643971 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:05.647820 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.647826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.647830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.648308 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.648692 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.648697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.648700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.648875 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.649486 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:05.649491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:05.649672 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.649803 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.650062 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:59:05.651301 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.635129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.12.182.124:40133)
0928 02:59:05.635272 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.635278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.635294 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:59:05.635392 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.635397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.635488 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.635571 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.636570 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.637789 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.639510 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.640214 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.640318 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.644153 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.644159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.644164 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.644195 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.647687 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.648431 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.648436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.648440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.648580 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.648994 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.648998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.649001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.649300 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.649431 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.650767 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.650981 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:59:05.652856 19183 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.182.124:40133: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2"
member_type: NON_VOTER
last_known_addr {
host: "127.12.182.124"
port: 40133
}
attrs {
promote: true
}
}
dest_uuid: "1cba5416a93846beb848f485afb759a0"
cas_config_opid_index: 5
I20260928 02:59:05.653919 19183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } }
I20260928 02:59:05.655767 19742 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2b6b0674db54a908303ea35bce9f8a2 (127.12.182.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:05.681740 19742 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.182.124:40133
I20260928 02:59:05.682392 19746 master.cc:537] Master@127.12.182.124:40133 shutting down...
I20260928 02:59:05.682638 19873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:05.684849 19792 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 LEARNER]: Refusing update from remote peer 1cba5416a93846beb848f485afb759a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:59:05.685530 19742 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:59:05.688117 19792 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2b6b0674db54a908303ea35bce9f8a2 (127.12.182.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:05.689957 19929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:05.690375 19929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:05.692204 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: NON_VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: true } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:05.692494 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:59:05.693146 19930 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0: attempting to promote NON_VOTER fbe76f76ea1b4f9b97ccab85094b50d7 to VOTER
I20260928 02:59:05.693795 19930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } }
I20260928 02:59:05.695779 19792 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 LEARNER]: Refusing update from remote peer 1cba5416a93846beb848f485afb759a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:59:05.696324 19930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 02:59:05.698611 19930 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fbe76f76ea1b4f9b97ccab85094b50d7 (127.12.182.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:05.699250 19791 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fbe76f76ea1b4f9b97ccab85094b50d7 (127.12.182.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:05.701978 19873 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 FOLLOWER]: Refusing update from remote peer 1cba5416a93846beb848f485afb759a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:59:05.703771 19930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:05.704550 19930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:05.704205 19928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:05.704794 19928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:05.710567 19746 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:05.710990 19746 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:05.711229 19746 tablet_replica.cc:330] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: stopping tablet replica
I20260928 02:59:05.714380 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:05.714612 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:59:05.718614 19746 master.cc:552] Master@127.12.182.124:40133 shutdown complete.
I20260928 02:59:05.718745 19746 master_runner.cc:305] Clearing existing system tablet
I20260928 02:59:05.719287 19746 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:05.719439 19746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260928 02:59:05.724053 19168 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 -> Peer b2b6b0674db54a908303ea35bce9f8a2 (127.12.182.124:40133): Couldn't send request to peer b2b6b0674db54a908303ea35bce9f8a2. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:59:05.728398 19746 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.008s sys 0.000s
I20260928 02:59:05.730017 19935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:05.730369 19746 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:05.730968 19746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:59:05.731122 19746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal
uuid: "b2b6b0674db54a908303ea35bce9f8a2"
format_stamp: "Formatted at 2026-09-28 02:59:05 on dist-test-slave-k5rg"
server_key: "fdc42b28597d9af6cffb51d72fb9dea2"
server_key_iv: "0f1c0f798d83c223a1e42dd9d05819ed"
server_key_version: "encryptionkey@0"
I20260928 02:59:05.736132 19746 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:05.743463 19746 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:59:05.743680 19746 log.cc:1196] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:59:05.744202 19746 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Deleting consensus metadata
I20260928 02:59:05.744445 19746 master_runner.cc:315] Copying system tablet from 127.12.182.126:33977
I20260928 02:59:05.747164 19746 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.12.182.126:33977
I20260928 02:59:05.761174 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 19245
I20260928 02:59:05.765388 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.748319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.12.182.126:33977)
0928 02:59:05.748554 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:05.748560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:05.748583 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:59:05.748780 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:05.748787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:05.748869 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:05.748967 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:05.750272 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.751394 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.753369 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.754081 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.754208 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.756369 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:05.756376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:05.756381 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:05.756420 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:05.762910 (+ 6490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.763505 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.763511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.763515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.763649 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:05.764019 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:05.764024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:05.764028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:05.764286 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:05.764424 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:05.764802 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 02:59:05.765048 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 02:59:05.765422 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:05.747946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50322)
0928 02:59:05.748450 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:05.748474 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:05.748604 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:05.749199 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:05.749204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:05.749787 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:05.750106 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.750120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.751565 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.751570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.753189 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:05.753209 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:05.754231 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:05.754237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:05.754424 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:05.755090 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:05.755113 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:05.756167 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:05.763019 (+ 6852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.763025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.763030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.763395 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.763752 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:05.763757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:05.763761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:05.763922 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:05.764407 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:05.764412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:05.764571 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 02:59:05.764681 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:59:05.764981 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":641,"threads_started":1}
I20260928 02:59:05.765936 19204 tablet_copy_service.cc:140] P 1cba5416a93846beb848f485afb759a0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b2b6b0674db54a908303ea35bce9f8a2 ({username='oryx', principal='oryx/127.12.182.124@KRBTEST.COM'} at 127.0.0.1:50322)
I20260928 02:59:05.766062 19204 tablet_copy_service.cc:161] P 1cba5416a93846beb848f485afb759a0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b2b6b0674db54a908303ea35bce9f8a2 at {username='oryx', principal='oryx/127.12.182.124@KRBTEST.COM'} at 127.0.0.1:50322: session id = b2b6b0674db54a908303ea35bce9f8a2-00000000000000000000000000000000
I20260928 02:59:05.768366 19204 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:59:05.769955 19746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:59:05.776870 19746 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: tablet copy: Starting download of 0 data blocks...
I20260928 02:59:05.777223 19746 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: tablet copy: Starting download of 1 WAL segments...
I20260928 02:59:05.778405 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:45413
--local_ip_for_outbound_sockets=127.12.182.65
--tserver_master_addrs=127.12.182.126:33977
--webserver_port=45423
--webserver_interface=127.12.182.65
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab
--principal=oryx/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.182.126:33977,127.12.182.125:37419
--tserver_master_addrs=127.12.182.126:33977,127.12.182.125:37419,127.12.182.124:40133 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:59:05.780274 19746 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:59:05.784574 19204 tablet_copy_service.cc:342] P 1cba5416a93846beb848f485afb759a0: Request end of tablet copy session b2b6b0674db54a908303ea35bce9f8a2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.182.124@KRBTEST.COM'} at 127.0.0.1:50322
I20260928 02:59:05.784698 19204 tablet_copy_service.cc:433] P 1cba5416a93846beb848f485afb759a0: ending tablet copy session b2b6b0674db54a908303ea35bce9f8a2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b2b6b0674db54a908303ea35bce9f8a2
I20260928 02:59:05.796010 19746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:05.798630 19943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:05.802903 19944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:05.813259 19946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:05.813522 19746 server_base.cc:1034] running on GCE node
I20260928 02:59:05.814062 19746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:05.814698 19746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:05.815887 19746 hybrid_clock.cc:648] HybridClock initialized: now 1790564345815846 us; error 30 us; skew 500 ppm
I20260928 02:59:05.816152 19746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:05 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564345, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:05.818329 19746 init.cc:377] Logged in from keytab as oryx/127.12.182.124@KRBTEST.COM (short username oryx)
I20260928 02:59:05.819767 19746 webserver.cc:460] Webserver started at http://127.12.182.124:35915/ using document root <none> and password file <none>
I20260928 02:59:05.820093 19746 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:05.820195 19746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:05.823181 19746 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:59:05.825013 19953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:05.825459 19746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 02:59:05.825626 19746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal
uuid: "b2b6b0674db54a908303ea35bce9f8a2"
format_stamp: "Formatted at 2026-09-28 02:59:05 on dist-test-slave-k5rg"
server_key: "fdc42b28597d9af6cffb51d72fb9dea2"
server_key_iv: "0f1c0f798d83c223a1e42dd9d05819ed"
server_key_version: "encryptionkey@0"
I20260928 02:59:05.825883 19746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:05.841549 19746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:05.845536 19746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:05.866066 19746 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.124:40133
I20260928 02:59:05.866114 20004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.124:40133 every 8 connection(s)
I20260928 02:59:05.867136 19746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
I20260928 02:59:05.872481 20005 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:59:05.876298 20005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Bootstrap starting.
I20260928 02:59:05.893586 20005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:59:05.894114 20005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Bootstrap complete.
I20260928 02:59:05.895126 20005 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } }
I20260928 02:59:05.895437 20005 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2b6b0674db54a908303ea35bce9f8a2, State: Initialized, Role: LEARNER
I20260928 02:59:05.895687 20005 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } }
I20260928 02:59:05.896735 20005 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:59:05.896478 20008 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:05.896981 20008 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:59:05.908172 20019 catalog_manager.cc:1259] Loaded cluster ID: 2539384aab9c4d7db2df51240b308ee4
I20260928 02:59:05.908262 20019 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: loading cluster ID for follower catalog manager: success
I20260928 02:59:05.911646 20019 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: acquiring CA information for follower catalog manager: success
I20260928 02:59:05.914077 20019 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:59:06.124681 19942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:06.124935 19942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:06.125052 19942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:06.132169 19942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:06.132266 19942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:06.132417 19942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:59:06.141891 19942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab
--principal=oryx/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:45413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=45423
--tserver_master_addrs=127.12.182.126:33977,127.12.182.125:37419,127.12.182.124:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:06.143119 19942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:06.144377 19942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:06.156848 20024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:06.156912 20025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:06.157361 20027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:06.158103 19942 server_base.cc:1034] running on GCE node
I20260928 02:59:06.158645 19942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:06.159519 19942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:06.160732 19942 hybrid_clock.cc:648] HybridClock initialized: now 1790564346160687 us; error 35 us; skew 500 ppm
I20260928 02:59:06.161052 19942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:06 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564346, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:06.164081 19942 init.cc:377] Logged in from keytab as oryx/127.12.182.65@KRBTEST.COM (short username oryx)
I20260928 02:59:06.165383 19942 webserver.cc:460] Webserver started at http://127.12.182.65:45423/ using document root <none> and password file <none>
I20260928 02:59:06.166096 19942 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:06.166249 19942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:06.172467 19942 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:59:06.172701 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.166765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:40133 (local address 127.0.0.1:35402)
0928 02:59:06.167238 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:06.167264 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:06.167415 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:06.167823 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:06.167826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:06.167843 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:06.168017 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.168025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.169714 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.169719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.170876 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.170890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.172108 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.172113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.172251 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.172278 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:59:06.172351 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:59:06.172450 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":223,"threads_started":1}
I20260928 02:59:06.172822 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.166880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.12.182.124:40133)
0928 02:59:06.167507 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:06.167513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:06.167533 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:59:06.167604 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:06.167608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:06.167697 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:06.167821 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:06.168140 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.169560 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.171057 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.171939 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.172093 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.172257 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:06.172392 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:59:06.172517 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":362,"threads_started":1}
I20260928 02:59:06.173532 19980 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 02:59:06.174113 20008 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cba5416a93846beb848f485afb759a0. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:06.174410 20008 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:59:06.175516 20036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:06.176812 19942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:59:06.176990 19942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "ad760c46b31745509ad8dc6f86e8f385"
format_stamp: "Formatted at 2026-09-28 02:59:03 on dist-test-slave-k5rg"
server_key: "5cc26b244a62b646d8feaeb381a1912a"
server_key_iv: "52d3b2828e5c7535fed6bfe965973dd5"
server_key_version: "encryptionkey@0"
I20260928 02:59:06.177934 19942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:06.200532 19942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:06.204455 19942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:06.204698 19942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:06.205686 19942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:06.206924 19942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:06.207018 19942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:06.207115 19942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:06.207183 19942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:06.235862 19942 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:45413
I20260928 02:59:06.235910 20148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:45413 every 8 connection(s)
I20260928 02:59:06.237165 19942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:59:06.238345 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 19942
I20260928 02:59:06.238691 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 19379
Sep 28 02:59:06 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564346, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.124@KRBTEST.COM
Sep 28 02:59:06 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564346, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.125@KRBTEST.COM
Sep 28 02:59:06 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564346, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
Sep 28 02:59:06 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.66@KRBTEST.COM
I20260928 02:59:06.272423 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.241558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:55079 (local address 127.12.182.124:40133)
0928 02:59:06.242103 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:06.242109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:06.256258 (+ 14149us) server_negotiation.cc:407] Connection header received
0928 02:59:06.256474 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:06.256479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:06.256571 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:06.256652 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:06.257853 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.258711 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.260527 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.261136 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.261227 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.266237 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:06.266242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:06.266245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:06.266272 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:06.268955 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.269820 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.269826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.269830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.269974 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.270453 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.270459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.270462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.270803 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:06.270929 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:06.271858 (+ 929us) server_negotiation.cc:299] Negotiation successful
0928 02:59:06.272070 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412}
I20260928 02:59:06.274705 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.240371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:37419 (local address 127.12.182.65:44491)
0928 02:59:06.248142 (+ 7771us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:06.248190 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:06.252283 (+ 4093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:06.256513 (+ 4230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:06.256518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:06.256947 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:06.257384 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.257396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.261116 (+ 3720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.261122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.262514 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.262528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.263474 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.263481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.263672 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.264225 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:06.264247 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:06.267166 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:06.271618 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.271623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.271627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.272073 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.272493 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.272498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.272501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.272645 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.273245 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:06.273250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:06.273421 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 02:59:06.273532 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:59:06.273819 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7450,"thread_start_us":15401,"threads_started":1}
I20260928 02:59:06.275375 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.241794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:44491 (local address 127.12.182.125:37419)
0928 02:59:06.255567 (+ 13773us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:06.255572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:06.255592 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:59:06.255640 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:06.255645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:06.255708 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:06.259705 (+ 3997us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:06.259911 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.260961 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.262655 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.263245 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.263961 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.267389 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:06.267396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:06.267400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:06.267449 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:06.271470 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.272221 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.272225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.272228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.272360 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.272782 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.272787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.272791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.273108 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:06.274697 (+ 1589us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:06.274826 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:59:06.275029 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13660,"thread_start_us":13562,"threads_started":1}
I20260928 02:59:06.275441 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.240371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:40133 (local address 127.12.182.65:55079)
0928 02:59:06.256159 (+ 15788us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:06.256175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:06.256344 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:06.256685 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:06.256693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:06.257384 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:06.257723 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.257736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.258863 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.258869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.260374 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.260389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.261598 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.261626 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.261905 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.262899 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:06.262934 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:06.266039 (+ 3105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:06.269102 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.269111 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.269127 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.269692 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.270078 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.270083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.270087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.270311 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.271029 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:06.271036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:06.271428 (+ 392us) client_negotiation.cc:769] Sending connection context
0928 02:59:06.272687 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0928 02:59:06.273188 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15651,"mutex_wait_us":3268,"thread_start_us":4000,"threads_started":1}
I20260928 02:59:06.276415 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.241478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:33939 (local address 127.12.182.126:33977)
0928 02:59:06.241684 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:06.241691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:06.252036 (+ 10345us) server_negotiation.cc:407] Connection header received
0928 02:59:06.252575 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:06.252582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:06.252673 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:06.252744 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:06.255284 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.256426 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.262631 (+ 6205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.263193 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.263291 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.268917 (+ 5626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:06.268924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:06.268928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:06.268968 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:06.272365 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.273424 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.273429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.273433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.273582 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.274076 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.274081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.274084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.274411 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:06.274536 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:06.275379 (+ 843us) server_negotiation.cc:299] Negotiation successful
0928 02:59:06.276092 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:59:06.276650 20151 heartbeater.cc:347] Connected to a master server at 127.12.182.125:37419
I20260928 02:59:06.276688 20150 heartbeater.cc:347] Connected to a master server at 127.12.182.124:40133
I20260928 02:59:06.276979 20150 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:06.276983 20151 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:06.277355 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.241121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.12.182.66:34761)
0928 02:59:06.255406 (+ 14285us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:06.255414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:06.255440 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:59:06.255504 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:06.255512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:06.255719 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:06.257791 (+ 2072us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:06.258036 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.259517 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.261442 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.262239 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.262358 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.268291 (+ 5933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:06.268305 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:06.268320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:06.268358 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:06.272229 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.273018 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.273025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.273029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.273165 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.273581 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.273586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.273590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.273996 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:06.275250 (+ 1254us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:06.275401 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:59:06.276571 (+ 1170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14123,"thread_start_us":13949,"threads_started":1}
I20260928 02:59:06.277522 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.241032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.65:33939)
0928 02:59:06.251937 (+ 10905us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:06.251954 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:06.252319 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:06.253178 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:06.253190 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:06.254214 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:06.255081 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.255107 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.257509 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.257514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.259562 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.259575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.263585 (+ 4010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.263595 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.263759 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.264380 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:06.264405 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:06.268704 (+ 4299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:06.272751 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.272755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.272758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.273291 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.273744 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.273749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.273753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.273948 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.274610 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:06.274614 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:06.274771 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 02:59:06.276784 (+ 2013us) client_negotiation.cc:240] Negotiation successful
0928 02:59:06.276991 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10784,"thread_start_us":11012,"threads_started":1}
I20260928 02:59:06.278044 20150 heartbeater.cc:511] Master 127.12.182.124:40133 requested a full tablet report, sending...
I20260928 02:59:06.278069 20151 heartbeater.cc:511] Master 127.12.182.125:37419 requested a full tablet report, sending...
I20260928 02:59:06.278659 20149 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33977
I20260928 02:59:06.278786 20149 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:06.279022 19502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:06.279104 20149 heartbeater.cc:511] Master 127.12.182.126:33977 requested a full tablet report, sending...
I20260928 02:59:06.280100 19183 ts_manager.cc:194] Re-registered known tserver with Master: ad760c46b31745509ad8dc6f86e8f385 (127.12.182.65:45413)
I20260928 02:59:06.280310 19782 ts_manager.cc:194] Registered new tserver with Master: ad760c46b31745509ad8dc6f86e8f385 (127.12.182.65:45413)
I20260928 02:59:06.281045 19183 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:33939
I20260928 02:59:06.283020 19970 ts_manager.cc:194] Registered new tserver with Master: ad760c46b31745509ad8dc6f86e8f385 (127.12.182.65:45413)
I20260928 02:59:06.363829 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 19379
I20260928 02:59:06.376612 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:34761
--local_ip_for_outbound_sockets=127.12.182.66
--tserver_master_addrs=127.12.182.126:33977
--webserver_port=41827
--webserver_interface=127.12.182.66
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab
--principal=oryx/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.182.126:33977,127.12.182.125:37419
--tserver_master_addrs=127.12.182.126:33977,127.12.182.125:37419,127.12.182.124:40133 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:06.681159 20160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:06.681449 20160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:06.681591 20160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:06.688934 20160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:06.689031 20160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:06.689146 20160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:59:06.698457 20160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab
--principal=oryx/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:34761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=41827
--tserver_master_addrs=127.12.182.126:33977,127.12.182.125:37419,127.12.182.124:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:06.699734 20160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:06.700968 20160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:06.712903 20165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:06.712985 20166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:06.713014 20168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:06.714244 20160 server_base.cc:1034] running on GCE node
I20260928 02:59:06.714789 20160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:06.715608 20160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:06.716801 20160 hybrid_clock.cc:648] HybridClock initialized: now 1790564346716754 us; error 34 us; skew 500 ppm
I20260928 02:59:06.717093 20160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:59:06.718307 19929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: NON_VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: true } } }
I20260928 02:59:06.718536 19929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:06.719538 19929 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0: attempting to promote NON_VOTER b2b6b0674db54a908303ea35bce9f8a2 to VOTER
I20260928 02:59:06.720124 19929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } }
Sep 28 02:59:06 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564346, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:06.721190 20160 init.cc:377] Logged in from keytab as oryx/127.12.182.66@KRBTEST.COM (short username oryx)
I20260928 02:59:06.722146 19980 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 LEARNER]: Refusing update from remote peer 1cba5416a93846beb848f485afb759a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 02:59:06.722146 19791 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 FOLLOWER]: Refusing update from remote peer 1cba5416a93846beb848f485afb759a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 02:59:06.722754 19929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:59:06.723075 20171 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:59:06.726846 19929 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2b6b0674db54a908303ea35bce9f8a2 (127.12.182.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:06.727639 19980 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2b6b0674db54a908303ea35bce9f8a2 (127.12.182.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:06.731364 20172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:06.731423 19929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:06.731693 19929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:06.731674 20172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:06.732746 20179 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:06.733038 20179 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:59:06.734671 20160 webserver.cc:460] Webserver started at http://127.12.182.66:41827/ using document root <none> and password file <none>
I20260928 02:59:06.735162 19929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:06.735425 19929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:06.735453 20160 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:06.735599 20160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:06.735736 19791 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2b6b0674db54a908303ea35bce9f8a2 (127.12.182.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:06.738538 20181 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cba5416a93846beb848f485afb759a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:06.738895 20181 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:59:06.740995 20160 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.002s
I20260928 02:59:06.743562 20183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:06.744779 20160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:59:06.744925 20160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "f29a5bb2b0074c3fbd25a6be50134e22"
format_stamp: "Formatted at 2026-09-28 02:59:03 on dist-test-slave-k5rg"
server_key: "8b55db6ae022dc38f01c8288d01de835"
server_key_iv: "666f2e139386d48dbb60243d8b62a255"
server_key_version: "encryptionkey@0"
I20260928 02:59:06.745872 20160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:06.786242 20160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:06.791221 20160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:06.791522 20160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:06.793510 20160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:06.795195 20160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:06.795300 20160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:06.795418 20160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:06.795496 20160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:06.823832 20160 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:34761
I20260928 02:59:06.823917 20295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:34761 every 8 connection(s)
I20260928 02:59:06.825169 20160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:59:06.833644 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 20160
I20260928 02:59:06.833942 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 19512
Sep 28 02:59:06 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564346, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
Sep 28 02:59:06 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564346, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.125@KRBTEST.COM
Sep 28 02:59:06 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564346, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.124@KRBTEST.COM
Sep 28 02:59:06 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.67@KRBTEST.COM
I20260928 02:59:06.865195 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.828210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:49621 (local address 127.12.182.125:37419)
0928 02:59:06.828623 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:06.828628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:06.829423 (+ 795us) server_negotiation.cc:407] Connection header received
0928 02:59:06.832160 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:06.832166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:06.832265 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:06.832342 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:06.849141 (+ 16799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.850199 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.852202 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.852800 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.852894 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.858072 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:06.858078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:06.858082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:06.858123 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:06.861448 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.862423 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.862429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.862432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.862588 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.863100 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.863105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.863108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.863440 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:06.863581 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:06.864617 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0928 02:59:06.864832 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":194,"threads_started":1}
I20260928 02:59:06.866168 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.830530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:48961 (local address 127.12.182.124:40133)
0928 02:59:06.846042 (+ 15512us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:06.846049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:06.848320 (+ 2271us) server_negotiation.cc:407] Connection header received
0928 02:59:06.848382 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:06.848387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:06.848463 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:06.848614 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:06.851227 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.852228 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.854037 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.854664 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.854774 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.859649 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:06.859656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:06.859661 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:06.859698 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:06.862812 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.863768 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.863772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.863776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.863928 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.864379 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.864384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.864387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.864729 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:06.864862 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:06.865627 (+ 765us) server_negotiation.cc:299] Negotiation successful
0928 02:59:06.865835 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15376,"thread_start_us":15263,"threads_started":1}
I20260928 02:59:06.866760 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.827913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:54363 (local address 127.12.182.126:33977)
0928 02:59:06.840555 (+ 12642us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:06.840562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:06.846543 (+ 5981us) server_negotiation.cc:407] Connection header received
0928 02:59:06.846695 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:06.846700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:06.846780 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:06.846897 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:06.848490 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.849631 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.852202 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.852816 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.852894 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.858969 (+ 6075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:06.858976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:06.858981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:06.859022 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:06.862889 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.863796 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.863801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.863804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.863968 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.864424 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.864428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.864432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.864763 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:06.864896 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:06.866193 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0928 02:59:06.866416 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12461,"thread_start_us":12641,"threads_started":1}
I20260928 02:59:06.867729 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.830123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:40133 (local address 127.12.182.66:48961)
0928 02:59:06.848089 (+ 17966us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:06.848105 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:06.848225 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:06.849711 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:06.849715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:06.850451 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:06.851059 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.851073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.852376 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.852381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.853875 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.853889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.855005 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.855012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.855207 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.855773 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:06.855795 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:06.859389 (+ 3594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:06.862950 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.862955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.862959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.863637 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.864053 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.864062 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.864066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.864239 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.864920 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:06.864925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:06.865075 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 02:59:06.865231 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 02:59:06.867150 (+ 1919us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17839,"mutex_wait_us":1701,"thread_start_us":17707,"threads_started":1}
I20260928 02:59:06.867774 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.828428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:37419 (local address 127.12.182.66:49621)
0928 02:59:06.829280 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:06.829321 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:06.831840 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:06.832828 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:06.832839 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:06.847160 (+ 14321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:06.848972 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.848988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.850418 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.850423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.852048 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.852062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.853232 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.853240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.853576 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.854394 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:06.854425 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:06.857850 (+ 3425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:06.861724 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.861733 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.861749 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.862286 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.862729 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.862733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.862737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.862986 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.863674 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:06.863682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:06.864129 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 02:59:06.865675 (+ 1546us) client_negotiation.cc:240] Negotiation successful
0928 02:59:06.865920 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":1151,"spinlock_wait_cycles":20352,"thread_start_us":448,"threads_started":1}
I20260928 02:59:06.868841 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.828974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.66:54363)
0928 02:59:06.846434 (+ 17460us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:06.846452 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:06.846577 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:06.847091 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:06.847096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:06.847678 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:06.848287 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.848311 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.849811 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.849817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.852047 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:06.852062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.853232 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.853240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:06.853575 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.854515 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:06.854533 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:06.858739 (+ 4206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:06.863025 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.863029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.863033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.863674 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.864094 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:06.864098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:06.864101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:06.864282 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:06.864956 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:06.864961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:06.865103 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 02:59:06.865256 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 02:59:06.865760 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17333,"mutex_wait_us":110,"thread_start_us":17072,"threads_started":1}
I20260928 02:59:06.869148 20298 heartbeater.cc:347] Connected to a master server at 127.12.182.125:37419
I20260928 02:59:06.869221 20297 heartbeater.cc:347] Connected to a master server at 127.12.182.124:40133
I20260928 02:59:06.869556 20298 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:06.869560 20297 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:06.870182 20296 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33977
I20260928 02:59:06.870306 20296 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:06.870524 20297 heartbeater.cc:511] Master 127.12.182.124:40133 requested a full tablet report, sending...
I20260928 02:59:06.870553 20298 heartbeater.cc:511] Master 127.12.182.125:37419 requested a full tablet report, sending...
I20260928 02:59:06.870702 20296 heartbeater.cc:511] Master 127.12.182.126:33977 requested a full tablet report, sending...
I20260928 02:59:06.872008 19183 ts_manager.cc:194] Re-registered known tserver with Master: f29a5bb2b0074c3fbd25a6be50134e22 (127.12.182.66:34761)
I20260928 02:59:06.872157 19782 ts_manager.cc:194] Registered new tserver with Master: f29a5bb2b0074c3fbd25a6be50134e22 (127.12.182.66:34761)
I20260928 02:59:06.872821 19183 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:54363
I20260928 02:59:06.873553 19970 ts_manager.cc:194] Registered new tserver with Master: f29a5bb2b0074c3fbd25a6be50134e22 (127.12.182.66:34761)
I20260928 02:59:06.877528 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:06.835296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.12.182.67:42661)
0928 02:59:06.849259 (+ 13963us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:06.849265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:06.849435 (+ 170us) server_negotiation.cc:407] Connection header received
0928 02:59:06.849622 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:06.849633 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:06.849880 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:06.850052 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:06.852061 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.853612 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.856183 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:06.856832 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:06.856936 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:06.862592 (+ 5656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:06.862605 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:06.862623 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:06.862667 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:06.874694 (+ 12027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.875551 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.875559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.875563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.875708 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:06.876025 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:06.876030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:06.876033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:06.876432 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:06.876687 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:06.876897 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:59:06.877137 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13719,"thread_start_us":13590,"threads_started":1}
I20260928 02:59:06.878700 19635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:06.958890 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 19512
I20260928 02:59:06.971413 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:42661
--local_ip_for_outbound_sockets=127.12.182.67
--tserver_master_addrs=127.12.182.126:33977
--webserver_port=35141
--webserver_interface=127.12.182.67
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab
--principal=oryx/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.182.126:33977,127.12.182.125:37419
--tserver_master_addrs=127.12.182.126:33977,127.12.182.125:37419,127.12.182.124:40133 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:07.279460 20309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:07.279700 20309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:07.279819 20309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:59:07.283475 20149 heartbeater.cc:503] Master 127.12.182.126:33977 was elected leader, sending a full tablet report...
W20260928 02:59:07.287621 20309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:07.287731 20309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:07.287880 20309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:59:07.297307 20309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab
--principal=oryx/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:42661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=35141
--tserver_master_addrs=127.12.182.126:33977,127.12.182.125:37419,127.12.182.124:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:07.298583 20309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:07.299870 20309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:07.311781 20314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:07.311916 20315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:07.312186 20317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:07.312814 20309 server_base.cc:1034] running on GCE node
I20260928 02:59:07.313392 20309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:07.314314 20309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:07.315508 20309 hybrid_clock.cc:648] HybridClock initialized: now 1790564347315462 us; error 35 us; skew 500 ppm
I20260928 02:59:07.315797 20309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:07 dist-test-slave-k5rg krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564347, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:07.319278 20309 init.cc:377] Logged in from keytab as oryx/127.12.182.67@KRBTEST.COM (short username oryx)
I20260928 02:59:07.320554 20309 webserver.cc:460] Webserver started at http://127.12.182.67:35141/ using document root <none> and password file <none>
I20260928 02:59:07.321175 20309 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:07.321305 20309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:07.326339 20309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:59:07.328825 20324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:07.330109 20309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:59:07.330252 20309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "86a5462cbbc34ec389f82d89abfe96c9"
format_stamp: "Formatted at 2026-09-28 02:59:04 on dist-test-slave-k5rg"
server_key: "57f88813160772d62e5fef7c866dbbcc"
server_key_iv: "99d5360204f6af275865a0396ee6c5f2"
server_key_version: "encryptionkey@0"
I20260928 02:59:07.331140 20309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:07.353214 20309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:07.357137 20309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:07.357388 20309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:07.358404 20309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:07.359762 20309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:07.359851 20309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:07.359953 20309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:07.360018 20309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:07.389356 20309 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:42661
I20260928 02:59:07.389416 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:42661 every 8 connection(s)
I20260928 02:59:07.390746 20309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:59:07.398602 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 20309
Sep 28 02:59:07 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564347, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.125@KRBTEST.COM
Sep 28 02:59:07 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564347, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.124@KRBTEST.COM
Sep 28 02:59:07 dist-test-slave-k5rg krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564347, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:07.417514 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.393753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:57821 (local address 127.12.182.124:40133)
0928 02:59:07.394187 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.394194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.396154 (+ 1960us) server_negotiation.cc:407] Connection header received
0928 02:59:07.397498 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.397503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.397588 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.397702 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:07.400131 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.401221 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.404318 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.404934 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.405027 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.410106 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:07.410113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:07.410118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:07.410153 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:07.413770 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.414742 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.414747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.414751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.414899 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.415470 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.415475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.415478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.415793 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:07.415915 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.416921 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.417143 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":686,"threads_started":1}
I20260928 02:59:07.417524 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.393670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:38755 (local address 127.12.182.125:37419)
0928 02:59:07.394617 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.394624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.395425 (+ 801us) server_negotiation.cc:407] Connection header received
0928 02:59:07.397430 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.397436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.397510 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.397598 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:07.400169 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.401324 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.404328 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.404960 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.405040 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.410401 (+ 5361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:07.410407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:07.410412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:07.410456 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:07.413715 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.414742 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.414747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.414751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.414894 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.415450 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.415455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.415458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.415796 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:07.415934 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.416931 (+ 997us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.417143 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":739,"threads_started":1}
I20260928 02:59:07.418563 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.393393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:41209 (local address 127.12.182.126:33977)
0928 02:59:07.393824 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.393830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.396075 (+ 2245us) server_negotiation.cc:407] Connection header received
0928 02:59:07.397434 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.397440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.397523 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.397623 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:07.400112 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.401227 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.404418 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.405070 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.405167 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.410572 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:07.410579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:07.410585 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:07.410628 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:07.414482 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.415274 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.415279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.415282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.415467 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.415920 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.415926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.415929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.416273 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:07.416414 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.416961 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.417204 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":382,"threads_started":1}
I20260928 02:59:07.419737 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.394103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:37419 (local address 127.12.182.67:38755)
0928 02:59:07.395266 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:07.395317 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:07.397159 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:07.397879 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:07.397885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:07.398667 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:07.399900 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.399915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.401576 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.401582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.404200 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.404210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.405371 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.405378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.405783 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.406775 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:07.406798 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:07.410189 (+ 3391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:07.413880 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:07.413889 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:07.413918 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:07.414612 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:07.415039 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:07.415044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:07.415047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:07.415312 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:07.416004 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:07.416012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:07.416495 (+ 483us) client_negotiation.cc:769] Sending connection context
0928 02:59:07.417799 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0928 02:59:07.418439 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":647,"spinlock_wait_cycles":157568,"thread_start_us":669,"threads_started":1}
I20260928 02:59:07.419834 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.394218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.67:41209)
0928 02:59:07.395950 (+ 1732us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:07.395971 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:07.397174 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:07.397869 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:07.397882 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:07.398905 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:07.399883 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.399916 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.401416 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.401421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.404210 (+ 2789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.404227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.405485 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.405492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.405771 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.406670 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:07.406692 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:07.410337 (+ 3645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:07.414643 (+ 4306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:07.414647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:07.414650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:07.415113 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:07.415607 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:07.415612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:07.415615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:07.415779 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:07.416495 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:07.416500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:07.416673 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 02:59:07.418997 (+ 2324us) client_negotiation.cc:240] Negotiation successful
0928 02:59:07.419282 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1584,"mutex_wait_us":894,"spinlock_wait_cycles":1408,"thread_start_us":495,"threads_started":1}
I20260928 02:59:07.419759 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.394115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:40133 (local address 127.12.182.67:57821)
0928 02:59:07.396079 (+ 1964us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:07.396093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:07.397174 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:07.397888 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:07.397892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:07.398905 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:07.399887 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.399908 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.401419 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.401423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.404165 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.404179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.405371 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.405378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.405778 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.406523 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:07.406552 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:07.409862 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:07.413884 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:07.413890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:07.413907 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:07.414612 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:07.415039 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:07.415044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:07.415058 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:07.415300 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:07.416004 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:07.416012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:07.416495 (+ 483us) client_negotiation.cc:769] Sending connection context
0928 02:59:07.417786 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0928 02:59:07.418322 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1846,"mutex_wait_us":761,"spinlock_wait_cycles":41088,"thread_start_us":1082,"threads_started":1}
I20260928 02:59:07.421252 20438 heartbeater.cc:347] Connected to a master server at 127.12.182.124:40133
I20260928 02:59:07.421345 20437 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33977
I20260928 02:59:07.421571 20437 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:07.421576 20438 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:07.421818 20439 heartbeater.cc:347] Connected to a master server at 127.12.182.125:37419
I20260928 02:59:07.421967 20439 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:07.422607 20438 heartbeater.cc:511] Master 127.12.182.124:40133 requested a full tablet report, sending...
I20260928 02:59:07.422713 20439 heartbeater.cc:511] Master 127.12.182.125:37419 requested a full tablet report, sending...
I20260928 02:59:07.422636 20437 heartbeater.cc:511] Master 127.12.182.126:33977 requested a full tablet report, sending...
I20260928 02:59:07.424350 19970 ts_manager.cc:194] Registered new tserver with Master: 86a5462cbbc34ec389f82d89abfe96c9 (127.12.182.67:42661)
I20260928 02:59:07.424639 19782 ts_manager.cc:194] Registered new tserver with Master: 86a5462cbbc34ec389f82d89abfe96c9 (127.12.182.67:42661)
I20260928 02:59:07.425676 19183 ts_manager.cc:194] Re-registered known tserver with Master: 86a5462cbbc34ec389f82d89abfe96c9 (127.12.182.67:42661)
I20260928 02:59:07.426749 19183 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:41209
I20260928 02:59:07.441993 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.425151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.12.182.126:33977)
0928 02:59:07.425765 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.425770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.425789 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:59:07.427158 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.427162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.427245 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.427312 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:07.429623 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.430382 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.432422 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.432988 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.433087 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.436404 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:07.436409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:07.436412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:07.436439 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:07.439427 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.440113 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.440117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.440120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.440250 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.440601 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.440606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.440609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.440853 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:07.441026 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.441412 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.441644 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509}
I20260928 02:59:07.442004 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.415495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.12.182.125:37419)
0928 02:59:07.417766 (+ 2271us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.417771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.423156 (+ 5385us) server_negotiation.cc:407] Connection header received
0928 02:59:07.426133 (+ 2977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.426137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.426225 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.426288 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:07.428966 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.429783 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.431604 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.432239 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.432356 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.436131 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:07.436138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:07.436143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:07.436180 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:07.438814 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.439459 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.439464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.439468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.439610 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.439970 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.439976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.439979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.440345 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:07.440582 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.441419 (+ 837us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.441655 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2176,"thread_start_us":7706,"threads_started":1}
I20260928 02:59:07.442054 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.424223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.12.182.124:40133)
0928 02:59:07.425334 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.425340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.425357 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:59:07.426855 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.426859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.426928 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.429510 (+ 2582us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:07.429650 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.430607 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.432339 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.432947 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.433066 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.435903 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:07.435909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:07.435914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:07.435941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:07.438870 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.439616 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.439621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.439625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.439770 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.440151 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.440156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.440160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.440455 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:07.440670 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.441521 (+ 851us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.441739 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002}
I20260928 02:59:07.446259 19183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:59:07.453073 19183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:59:07.496557 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.469296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:42661 (local address 127.0.0.1:59738)
0928 02:59:07.469937 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:07.469965 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:07.470095 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:07.485409 (+ 15314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:07.485415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:07.485450 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:07.485888 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.485901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.491034 (+ 5133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.491041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.493338 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.493356 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.495626 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.495632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.495822 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.495859 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:59:07.495962 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:59:07.496105 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":332,"threads_started":1}
I20260928 02:59:07.496560 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.469500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.12.182.67:42661)
0928 02:59:07.484680 (+ 15180us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.484691 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.484730 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:59:07.484848 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.484857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.485165 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.485389 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:07.487219 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.488837 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.493526 (+ 4689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.495429 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.495568 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.495799 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.496024 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.496201 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14978,"thread_start_us":15192,"threads_started":1}
I20260928 02:59:07.503862 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.469984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.0.0.1:38158)
0928 02:59:07.489415 (+ 19431us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:07.489433 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:07.493532 (+ 4099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:07.494747 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:07.494752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:07.494780 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:07.495214 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.495226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.499415 (+ 4189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.499421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.501347 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.501360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.503086 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.503092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.503249 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.503282 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:59:07.503384 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:59:07.503512 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19287,"thread_start_us":29537,"threads_started":1}
I20260928 02:59:07.503358 20372 tablet_service.cc:1467] Processing CreateTablet for tablet f6818a64807f4b0496cf774d2e857e89 (DEFAULT_TABLE table=test-table [id=8bad56ac66bb46908a475d3052f5cc32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:07.504185 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.470544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.12.182.65:45413)
0928 02:59:07.493859 (+ 23315us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.493870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.493902 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:59:07.494017 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.494025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.494306 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.494505 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:07.495424 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.499231 (+ 3807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.501527 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.502895 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.503017 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.503205 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.503478 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.503645 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20369,"thread_start_us":20232,"threads_started":1}
I20260928 02:59:07.504882 20372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6818a64807f4b0496cf774d2e857e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:07.507601 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.469987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:34761 (local address 127.0.0.1:58862)
0928 02:59:07.497110 (+ 27123us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:07.497130 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:07.497229 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:07.501580 (+ 4351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:07.501584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:07.501632 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:07.501785 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.501796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.503404 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.503417 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.504983 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.504994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.506858 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.506864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.507046 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.507078 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:59:07.507165 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:59:07.507290 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27021,"mutex_wait_us":734,"thread_start_us":18409,"threads_started":1}
I20260928 02:59:07.508327 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.470507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.12.182.66:34761)
0928 02:59:07.501095 (+ 30588us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.501103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.501129 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:59:07.501208 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.501213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.501384 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.501627 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:07.501934 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.503285 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.505127 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.506463 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.506587 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.506751 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.507937 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.508091 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30442,"thread_start_us":30348,"threads_started":1}
I20260928 02:59:07.508011 20084 tablet_service.cc:1467] Processing CreateTablet for tablet f6818a64807f4b0496cf774d2e857e89 (DEFAULT_TABLE table=test-table [id=8bad56ac66bb46908a475d3052f5cc32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:07.511029 20231 tablet_service.cc:1467] Processing CreateTablet for tablet f6818a64807f4b0496cf774d2e857e89 (DEFAULT_TABLE table=test-table [id=8bad56ac66bb46908a475d3052f5cc32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:07.511643 20084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6818a64807f4b0496cf774d2e857e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:07.512056 20231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6818a64807f4b0496cf774d2e857e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:07.516342 20469 tablet_bootstrap.cc:492] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: Bootstrap starting.
I20260928 02:59:07.519529 20469 tablet_bootstrap.cc:654] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:07.520442 20469 log.cc:824] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:07.522794 20469 tablet_bootstrap.cc:492] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: No bootstrap required, opened a new log
I20260928 02:59:07.523034 20469 ts_tablet_manager.cc:1397] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:59:07.526821 20472 tablet_bootstrap.cc:492] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: Bootstrap starting.
I20260928 02:59:07.526960 20471 tablet_bootstrap.cc:492] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: Bootstrap starting.
I20260928 02:59:07.529528 20471 tablet_bootstrap.cc:654] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:07.530685 20471 log.cc:824] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:07.530952 20472 tablet_bootstrap.cc:654] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:07.532039 20472 log.cc:824] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:07.531786 20469 raft_consensus.cc:348] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } }
I20260928 02:59:07.532198 20469 raft_consensus.cc:374] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:07.532299 20469 raft_consensus.cc:729] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a5462cbbc34ec389f82d89abfe96c9, State: Initialized, Role: FOLLOWER
I20260928 02:59:07.532979 20469 consensus_queue.cc:260] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } }
I20260928 02:59:07.534087 20437 heartbeater.cc:503] Master 127.12.182.126:33977 was elected leader, sending a full tablet report...
I20260928 02:59:07.534680 20469 ts_tablet_manager.cc:1428] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260928 02:59:07.534935 20472 tablet_bootstrap.cc:492] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: No bootstrap required, opened a new log
I20260928 02:59:07.535172 20471 tablet_bootstrap.cc:492] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: No bootstrap required, opened a new log
I20260928 02:59:07.535199 20472 ts_tablet_manager.cc:1397] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260928 02:59:07.535502 20471 ts_tablet_manager.cc:1397] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 02:59:07.545241 20471 raft_consensus.cc:348] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } }
I20260928 02:59:07.545241 20472 raft_consensus.cc:348] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } }
I20260928 02:59:07.545706 20472 raft_consensus.cc:374] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:07.545734 20471 raft_consensus.cc:374] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:07.545821 20472 raft_consensus.cc:729] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29a5bb2b0074c3fbd25a6be50134e22, State: Initialized, Role: FOLLOWER
I20260928 02:59:07.545863 20471 raft_consensus.cc:729] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad760c46b31745509ad8dc6f86e8f385, State: Initialized, Role: FOLLOWER
I20260928 02:59:07.546446 20471 consensus_queue.cc:260] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } }
I20260928 02:59:07.546437 20472 consensus_queue.cc:260] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } }
I20260928 02:59:07.547452 20296 heartbeater.cc:503] Master 127.12.182.126:33977 was elected leader, sending a full tablet report...
I20260928 02:59:07.548218 20472 ts_tablet_manager.cc:1428] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
I20260928 02:59:07.549621 20471 ts_tablet_manager.cc:1428] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260928 02:59:07.555501 20476 raft_consensus.cc:482] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:59:07.555688 20476 raft_consensus.cc:504] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } }
I20260928 02:59:07.557017 20476 leader_election.cc:288] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a5462cbbc34ec389f82d89abfe96c9 (127.12.182.67:42661), ad760c46b31745509ad8dc6f86e8f385 (127.12.182.65:45413)
I20260928 02:59:07.562767 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.557304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:45845 (local address 127.12.182.67:42661)
0928 02:59:07.557537 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.557544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.557833 (+ 289us) server_negotiation.cc:407] Connection header received
0928 02:59:07.558008 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.558013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.558087 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.558173 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:07.558668 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.559406 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.561126 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.561953 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.562097 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.562171 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.562411 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.562536 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:59:07.562888 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.557197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:42661 (local address 127.12.182.66:45845)
0928 02:59:07.557744 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:07.557759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:07.557881 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:07.558218 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:07.558224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:07.558255 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:07.558532 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.558542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.559532 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.559535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.560984 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.560996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.562123 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.562129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.562274 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.562297 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:59:07.562385 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:59:07.562511 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":1,"thread_start_us":271,"threads_started":1}
I20260928 02:59:07.563234 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.557942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:47391 (local address 127.12.182.65:45413)
0928 02:59:07.558414 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.558420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.558462 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:59:07.558637 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.558642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.558697 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.558762 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:07.559251 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.559990 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.561657 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.562395 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.562536 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.562603 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.562880 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.563008 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260928 02:59:07.563279 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.557867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.12.182.66:47391)
0928 02:59:07.558380 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:07.558399 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:07.558531 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:07.558786 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:07.558790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:07.558812 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:07.559114 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.559123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.560128 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.560133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.561499 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:07.561511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.562589 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.562594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:07.562744 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.562770 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:59:07.562850 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:59:07.562972 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":245,"threads_started":1}
I20260928 02:59:07.563797 20392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6818a64807f4b0496cf774d2e857e89" candidate_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a5462cbbc34ec389f82d89abfe96c9" is_pre_election: true
I20260928 02:59:07.564050 20104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6818a64807f4b0496cf774d2e857e89" candidate_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad760c46b31745509ad8dc6f86e8f385" is_pre_election: true
I20260928 02:59:07.564260 20392 raft_consensus.cc:2454] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29a5bb2b0074c3fbd25a6be50134e22 in term 0.
I20260928 02:59:07.564491 20104 raft_consensus.cc:2454] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29a5bb2b0074c3fbd25a6be50134e22 in term 0.
I20260928 02:59:07.564885 20187 leader_election.cc:302] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86a5462cbbc34ec389f82d89abfe96c9, f29a5bb2b0074c3fbd25a6be50134e22; no voters:
I20260928 02:59:07.565300 20476 raft_consensus.cc:2781] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:59:07.565466 20476 raft_consensus.cc:482] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:59:07.565572 20476 raft_consensus.cc:3037] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:07.567667 20476 raft_consensus.cc:504] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } }
I20260928 02:59:07.568166 20476 leader_election.cc:288] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [CANDIDATE]: Term 1 election: Requested vote from peers 86a5462cbbc34ec389f82d89abfe96c9 (127.12.182.67:42661), ad760c46b31745509ad8dc6f86e8f385 (127.12.182.65:45413)
I20260928 02:59:07.568583 20392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6818a64807f4b0496cf774d2e857e89" candidate_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a5462cbbc34ec389f82d89abfe96c9"
I20260928 02:59:07.568598 20104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6818a64807f4b0496cf774d2e857e89" candidate_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad760c46b31745509ad8dc6f86e8f385"
I20260928 02:59:07.568739 20104 raft_consensus.cc:3037] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:07.568751 20392 raft_consensus.cc:3037] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:07.570806 20104 raft_consensus.cc:2454] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29a5bb2b0074c3fbd25a6be50134e22 in term 1.
I20260928 02:59:07.570958 20392 raft_consensus.cc:2454] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29a5bb2b0074c3fbd25a6be50134e22 in term 1.
I20260928 02:59:07.571151 20187 leader_election.cc:302] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad760c46b31745509ad8dc6f86e8f385, f29a5bb2b0074c3fbd25a6be50134e22; no voters:
I20260928 02:59:07.571434 20476 raft_consensus.cc:2781] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:07.571842 20476 raft_consensus.cc:686] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 1 LEADER]: Becoming Leader. State: Replica: f29a5bb2b0074c3fbd25a6be50134e22, State: Running, Role: LEADER
I20260928 02:59:07.572319 20476 consensus_queue.cc:237] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } }
I20260928 02:59:07.577257 19184 catalog_manager.cc:5495] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 reported cstate change: term changed from 0 to 1, leader changed from <none> to f29a5bb2b0074c3fbd25a6be50134e22 (127.12.182.66). New cstate: current_term: 1 leader_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } health_report { overall_health: HEALTHY } } }
W20260928 02:59:07.578719 20299 tablet.cc:2369] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:59:07.600835 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.596045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.12.182.66:34761)
0928 02:59:07.596209 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.596214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.596243 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:59:07.596318 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.596322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.596420 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.596525 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:07.596816 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.597530 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.598844 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.599287 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.599426 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.599838 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:07.600385 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:07.600493 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.600575 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.600634 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:59:07.608038 20104 raft_consensus.cc:1261] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385 [term 1 FOLLOWER]: Refusing update from remote peer f29a5bb2b0074c3fbd25a6be50134e22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:07.608070 20392 raft_consensus.cc:1261] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9 [term 1 FOLLOWER]: Refusing update from remote peer f29a5bb2b0074c3fbd25a6be50134e22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:07.608837 20480 consensus_queue.cc:1035] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:07.609141 20476 consensus_queue.cc:1035] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:07.621269 20484 mvcc.cc:205] Tried to move back new op lower bound from 7334151567794192384 to 7334151567660081152. Current Snapshot: MvccSnapshot[applied={T|T < 7334151567794192384}]
I20260928 02:59:07.623935 20485 mvcc.cc:205] Tried to move back new op lower bound from 7334151567794192384 to 7334151567660081152. Current Snapshot: MvccSnapshot[applied={T|T < 7334151567794192384}]
I20260928 02:59:07.635165 19184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:07.635450 19184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:07.641454 19184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cba5416a93846beb848f485afb759a0: Sending DeleteTablet for 3 replicas of tablet f6818a64807f4b0496cf774d2e857e89
I20260928 02:59:07.642853 20372 tablet_service.cc:1514] Processing DeleteTablet for tablet f6818a64807f4b0496cf774d2e857e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:07 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:59738
I20260928 02:59:07.642930 20231 tablet_service.cc:1514] Processing DeleteTablet for tablet f6818a64807f4b0496cf774d2e857e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:07 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:58862
I20260928 02:59:07.643997 20084 tablet_service.cc:1514] Processing DeleteTablet for tablet f6818a64807f4b0496cf774d2e857e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:07 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:38158
I20260928 02:59:07.649717 20495 tablet_replica.cc:330] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: stopping tablet replica
I20260928 02:59:07.650175 20495 raft_consensus.cc:2229] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:07.653803 20495 raft_consensus.cc:2258] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:07.655958 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 19152
I20260928 02:59:07.657059 19214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:07.657226 20493 tablet_replica.cc:330] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: stopping tablet replica
I20260928 02:59:07.657645 20493 raft_consensus.cc:2229] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:07.658094 20493 raft_consensus.cc:2258] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:07.658661 20495 ts_tablet_manager.cc:1905] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:07.660086 20493 ts_tablet_manager.cc:1905] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:07.666862 20493 ts_tablet_manager.cc:1918] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:07.667035 20493 log.cc:1196] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/f6818a64807f4b0496cf774d2e857e89
I20260928 02:59:07.667568 20493 ts_tablet_manager.cc:1939] T f6818a64807f4b0496cf774d2e857e89 P 86a5462cbbc34ec389f82d89abfe96c9: Deleting consensus metadata
I20260928 02:59:07.669560 20494 tablet_replica.cc:330] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: stopping tablet replica
I20260928 02:59:07.669993 20494 raft_consensus.cc:2229] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:59:07.670710 20494 raft_consensus.cc:2258] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:07.672437 20494 ts_tablet_manager.cc:1905] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:07.673127 20495 ts_tablet_manager.cc:1918] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:07.673285 20495 log.cc:1196] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/f6818a64807f4b0496cf774d2e857e89
I20260928 02:59:07.673794 20495 ts_tablet_manager.cc:1939] T f6818a64807f4b0496cf774d2e857e89 P ad760c46b31745509ad8dc6f86e8f385: Deleting consensus metadata
I20260928 02:59:07.678977 20494 ts_tablet_manager.cc:1918] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:07.679139 20494 log.cc:1196] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/f6818a64807f4b0496cf774d2e857e89
I20260928 02:59:07.679724 20494 ts_tablet_manager.cc:1939] T f6818a64807f4b0496cf774d2e857e89 P f29a5bb2b0074c3fbd25a6be50134e22: Deleting consensus metadata
I20260928 02:59:07.687237 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.645920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.12.182.125:37419)
0928 02:59:07.646074 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.646080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.654774 (+ 8694us) server_negotiation.cc:407] Connection header received
0928 02:59:07.656179 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.656186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.656266 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.656336 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:07.661824 (+ 5488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.662956 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.676663 (+ 13707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.677357 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.677514 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.681726 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:07.681732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:07.681736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:07.681783 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:07.684501 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.685140 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.685144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.685147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.685272 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.685747 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.685757 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.685760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.686100 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:07.686325 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.686663 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.686872 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:59:07.699370 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.654882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.12.182.124:40133)
0928 02:59:07.655059 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.655066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.687223 (+ 32157us) server_negotiation.cc:407] Connection header received
0928 02:59:07.687857 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.687863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.687946 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.688013 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:07.689455 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.690213 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.691376 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.691771 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.691875 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.694312 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:07.694317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:07.694320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:07.694360 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:07.696742 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.697305 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.697311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.697314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.697444 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.697856 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.697861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.697864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.698173 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:07.698389 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.698764 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.698970 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:59:07.845243 19168 catalog_manager.cc:4841] TS f29a5bb2b0074c3fbd25a6be50134e22 (127.12.182.66:34761): tablet f6818a64807f4b0496cf774d2e857e89 (table test-table [id=8bad56ac66bb46908a475d3052f5cc32]) successfully deleted
I20260928 02:59:07.846396 19171 catalog_manager.cc:4841] TS ad760c46b31745509ad8dc6f86e8f385 (127.12.182.65:45413): tablet f6818a64807f4b0496cf774d2e857e89 (table test-table [id=8bad56ac66bb46908a475d3052f5cc32]) successfully deleted
I20260928 02:59:07.846624 19171 catalog_manager.cc:4841] TS 86a5462cbbc34ec389f82d89abfe96c9 (127.12.182.67:42661): tablet f6818a64807f4b0496cf774d2e857e89 (table test-table [id=8bad56ac66bb46908a475d3052f5cc32]) successfully deleted
I20260928 02:59:07.846853 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 19152
I20260928 02:59:07.907387 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.880510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.12.182.124:40133)
0928 02:59:07.880720 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.880726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.894046 (+ 13320us) server_negotiation.cc:407] Connection header received
0928 02:59:07.894292 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.894299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.894375 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.894448 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:07.896482 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.897217 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.898522 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.898955 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.899065 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.902088 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:07.902094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:07.902099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:07.902136 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:07.904539 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.905424 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.905430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.905434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.905585 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.905985 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.905990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.905993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.906336 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:07.906572 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.906851 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.907100 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:59:07.908838 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:07.874423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.12.182.125:37419)
0928 02:59:07.874653 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:07.874660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:07.881821 (+ 7161us) server_negotiation.cc:407] Connection header received
0928 02:59:07.882184 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:07.882190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:07.882280 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:07.882353 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:07.889743 (+ 7390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.890965 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.894175 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:07.894819 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:07.894919 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:07.901414 (+ 6495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:07.901419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:07.901423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:07.901459 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:07.904669 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.905704 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.905709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.905712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.905859 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:07.906954 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:07.906959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:07.906963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:07.907305 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:07.907614 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:07.908164 (+ 550us) server_negotiation.cc:299] Negotiation successful
0928 02:59:07.908378 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:59:08.625931 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:08.625114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.65:33835)
0928 02:59:08.625582 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:08.625743 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":233,"threads_started":1}
I20260928 02:59:08.626111 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:08.625230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.66:41819)
0928 02:59:08.625805 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:08.625940 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"threads_started":1}
W20260928 02:59:08.626367 20149 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:33977 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
W20260928 02:59:08.626653 20296 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:33977 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
I20260928 02:59:08.629352 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:08.628596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.67:45381)
0928 02:59:08.628980 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:08.629112 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":188,"threads_started":1}
W20260928 02:59:08.629705 20437 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:33977 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
I20260928 02:59:09.141811 20508 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cba5416a93846beb848f485afb759a0)
I20260928 02:59:09.142058 20508 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } }
I20260928 02:59:09.143318 20508 leader_election.cc:288] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cba5416a93846beb848f485afb759a0 (127.12.182.126:33977), fbe76f76ea1b4f9b97ccab85094b50d7 (127.12.182.125:37419)
I20260928 02:59:09.144570 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.143406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50738)
0928 02:59:09.143950 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.144268 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":258,"threads_started":1}
W20260928 02:59:09.144902 19957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
I20260928 02:59:09.146098 20512 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cba5416a93846beb848f485afb759a0)
I20260928 02:59:09.146466 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.146079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50752)
0928 02:59:09.146247 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.146329 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:59:09.146373 20512 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } }
W20260928 02:59:09.146672 19957 leader_election.cc:334] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1cba5416a93846beb848f485afb759a0 (127.12.182.126:33977): Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
I20260928 02:59:09.147837 20512 leader_election.cc:288] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cba5416a93846beb848f485afb759a0 (127.12.182.126:33977), b2b6b0674db54a908303ea35bce9f8a2 (127.12.182.124:40133)
I20260928 02:59:09.152405 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.143519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:37419 (local address 127.0.0.1:44218)
0928 02:59:09.144309 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.144325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:09.144457 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:09.145641 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:09.145649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:09.145676 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:09.145994 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.146008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.147268 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.147274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.149499 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.149516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.151606 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.151612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.151800 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.151835 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:59:09.151918 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:59:09.152057 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":511,"threads_started":1}
I20260928 02:59:09.153342 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.143678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.12.182.125:37419)
0928 02:59:09.145286 (+ 1608us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.145291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.145312 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:59:09.145387 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.145393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.145473 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.145574 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:09.146176 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.147150 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.149700 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.151257 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.151417 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.151501 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.152810 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.152959 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1494,"thread_start_us":1259,"threads_started":1}
I20260928 02:59:09.155535 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.147887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50762)
0928 02:59:09.155041 (+ 7154us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.155216 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7010,"thread_start_us":6886,"threads_started":1}
W20260928 02:59:09.155830 19769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
I20260928 02:59:09.156018 19791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2b6b0674db54a908303ea35bce9f8a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" is_pre_election: true
I20260928 02:59:09.156471 19791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b6b0674db54a908303ea35bce9f8a2 in term 1.
I20260928 02:59:09.157050 19955 leader_election.cc:302] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b2b6b0674db54a908303ea35bce9f8a2, fbe76f76ea1b4f9b97ccab85094b50d7; no voters: 1cba5416a93846beb848f485afb759a0
I20260928 02:59:09.157413 20508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:59:09.157497 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.157137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50770)
0928 02:59:09.157308 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.157381 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 02:59:09.157546 20508 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cba5416a93846beb848f485afb759a0)
I20260928 02:59:09.157680 20508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 1 FOLLOWER]: Advancing to term 2
W20260928 02:59:09.157637 19769 leader_election.cc:334] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1cba5416a93846beb848f485afb759a0 (127.12.182.126:33977): Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
I20260928 02:59:09.159876 20508 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } }
I20260928 02:59:09.160589 20508 leader_election.cc:288] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [CANDIDATE]: Term 2 election: Requested vote from peers 1cba5416a93846beb848f485afb759a0 (127.12.182.126:33977), fbe76f76ea1b4f9b97ccab85094b50d7 (127.12.182.125:37419)
I20260928 02:59:09.161088 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.160716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50778)
0928 02:59:09.160885 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.160967 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:59:09.161080 19791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2b6b0674db54a908303ea35bce9f8a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7"
I20260928 02:59:09.161316 19791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:59:09.161859 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.148019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:40133 (local address 127.0.0.1:35420)
0928 02:59:09.155635 (+ 7616us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.155651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:09.155768 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:09.156107 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:09.156112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:09.156139 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:09.156637 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.156652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.158095 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.158101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.159344 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.159359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.161069 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.161076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.161256 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.161299 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:59:09.161404 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:59:09.161525 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7507,"thread_start_us":7350,"threads_started":1}
I20260928 02:59:09.161896 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.148345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.12.182.124:40133)
0928 02:59:09.155400 (+ 7055us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.155405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.155718 (+ 313us) server_negotiation.cc:407] Connection header received
0928 02:59:09.155894 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.155899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.155966 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.156058 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:09.156818 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.157982 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.159520 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.160755 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.160923 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.160999 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.161429 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.161545 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6939,"thread_start_us":6819,"threads_started":1}
I20260928 02:59:09.162616 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.162317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50790)
0928 02:59:09.162422 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.162510 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 02:59:09.162485 19980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b2b6b0674db54a908303ea35bce9f8a2" is_pre_election: true
W20260928 02:59:09.162801 19957 leader_election.cc:334] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1cba5416a93846beb848f485afb759a0 (127.12.182.126:33977): Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
I20260928 02:59:09.162882 19980 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fbe76f76ea1b4f9b97ccab85094b50d7 in current term 2: Already voted for candidate b2b6b0674db54a908303ea35bce9f8a2 in this term.
I20260928 02:59:09.163472 19766 leader_election.cc:302] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe76f76ea1b4f9b97ccab85094b50d7; no voters: 1cba5416a93846beb848f485afb759a0, b2b6b0674db54a908303ea35bce9f8a2
I20260928 02:59:09.164175 19791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b6b0674db54a908303ea35bce9f8a2 in term 2.
I20260928 02:59:09.164328 20512 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:59:09.164507 19955 leader_election.cc:302] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b2b6b0674db54a908303ea35bce9f8a2, fbe76f76ea1b4f9b97ccab85094b50d7; no voters: 1cba5416a93846beb848f485afb759a0
I20260928 02:59:09.164732 20508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:59:09.165125 20508 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [term 2 LEADER]: Becoming Leader. State: Replica: b2b6b0674db54a908303ea35bce9f8a2, State: Running, Role: LEADER
I20260928 02:59:09.165500 20508 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } }
I20260928 02:59:09.167727 20518 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2b6b0674db54a908303ea35bce9f8a2. Latest consensus state: current_term: 2 leader_uuid: "b2b6b0674db54a908303ea35bce9f8a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:09.168037 20518 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:09.168604 20520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:59:09.171903 20520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:59:09.172663 20520 catalog_manager.cc:1259] Loaded cluster ID: 2539384aab9c4d7db2df51240b308ee4
I20260928 02:59:09.172729 20520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:59:09.173818 20520 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:59:09.174548 20520 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Loaded TSK: 0
I20260928 02:59:09.175542 20520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:59:09.228672 19970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:59:09.237867 19970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:59:09.241889 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.241512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50808)
0928 02:59:09.241692 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.241766 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:59:09.241951 19791 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [term 2 FOLLOWER]: Refusing update from remote peer b2b6b0674db54a908303ea35bce9f8a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 02:59:09.242712 20518 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 02:59:09.242887 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.242497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50824)
0928 02:59:09.242653 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.242738 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 02:59:09.243031 19957 consensus_peers.cc:487] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 -> Peer 1cba5416a93846beb848f485afb759a0 (127.12.182.126:33977): Couldn't send request to peer 1cba5416a93846beb848f485afb759a0. Status: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:59:09.245316 20512 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2b6b0674db54a908303ea35bce9f8a2. Latest consensus state: current_term: 2 leader_uuid: "b2b6b0674db54a908303ea35bce9f8a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:09.245536 20512 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:59:09.246732 20518 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b2b6b0674db54a908303ea35bce9f8a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:09.247056 20518 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:09.247326 20512 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b2b6b0674db54a908303ea35bce9f8a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1cba5416a93846beb848f485afb759a0" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33977 } } peers { permanent_uuid: "fbe76f76ea1b4f9b97ccab85094b50d7" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "b2b6b0674db54a908303ea35bce9f8a2" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 40133 } attrs { promote: false } } }
I20260928 02:59:09.247556 20512 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe76f76ea1b4f9b97ccab85094b50d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:59:09.260380 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.253826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:34761 (local address 127.0.0.1:58880)
0928 02:59:09.254186 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.254203 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:09.254301 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:09.254998 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:09.255003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:09.255024 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:09.255222 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.255237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.256362 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.256367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.258141 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.258154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.259597 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.259603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.259816 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.259853 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:59:09.259955 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:59:09.260089 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260928 02:59:09.261358 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.253936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.12.182.66:34761)
0928 02:59:09.254750 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.254754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.254768 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:59:09.254824 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.254827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.254884 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.254986 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:09.255356 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.256240 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.258306 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.259457 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.259581 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.259682 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.259997 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.260135 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":637,"threads_started":1}
I20260928 02:59:09.262148 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.254229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.0.0.1:38164)
0928 02:59:09.254518 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.254534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:09.254640 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:09.255607 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:09.255611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:09.255632 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:09.255981 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.255994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.257548 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.257554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.259618 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.259632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.261080 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.261087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.261275 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.261311 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:59:09.261485 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:59:09.261865 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260928 02:59:09.262406 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.254839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.12.182.65:45413)
0928 02:59:09.255287 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.255292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.255310 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:59:09.255376 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.255380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.255462 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.256123 (+ 661us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:09.256291 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.257397 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.259782 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.260841 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.260982 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.261085 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.261696 (+ 611us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.261839 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":844,"threads_started":1}
I20260928 02:59:09.262416 20231 tablet_service.cc:1467] Processing CreateTablet for tablet b68d0e54622b42f0b7c645f3a142156c (DEFAULT_TABLE table=test-table [id=9bb4df3ef46e4e01980680e186465d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:09.263031 20231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b68d0e54622b42f0b7c645f3a142156c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:09.263286 20084 tablet_service.cc:1467] Processing CreateTablet for tablet b68d0e54622b42f0b7c645f3a142156c (DEFAULT_TABLE table=test-table [id=9bb4df3ef46e4e01980680e186465d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:09.263739 20084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b68d0e54622b42f0b7c645f3a142156c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:09.272719 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.255208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:42661 (local address 127.0.0.1:59740)
0928 02:59:09.255698 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.255715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:09.255821 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:09.266310 (+ 10489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:09.266315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:09.266344 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:09.266644 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.266654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.267991 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.268004 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.270026 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.270043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.271884 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.271891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.272091 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.272127 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:59:09.272214 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:59:09.272354 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":531,"threads_started":1}
I20260928 02:59:09.273187 20530 tablet_bootstrap.cc:492] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22: Bootstrap starting.
I20260928 02:59:09.273389 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.255600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.12.182.67:42661)
0928 02:59:09.265982 (+ 10382us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.265988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.266008 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:59:09.266078 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.266083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.266167 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.266270 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:09.266805 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.267837 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.270225 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.271493 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.271618 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.271734 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.272928 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.273101 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10250,"thread_start_us":10155,"threads_started":1}
I20260928 02:59:09.274099 20372 tablet_service.cc:1467] Processing CreateTablet for tablet b68d0e54622b42f0b7c645f3a142156c (DEFAULT_TABLE table=test-table [id=9bb4df3ef46e4e01980680e186465d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:09.274513 20372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b68d0e54622b42f0b7c645f3a142156c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:09.274605 20531 tablet_bootstrap.cc:492] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385: Bootstrap starting.
I20260928 02:59:09.274991 20530 tablet_bootstrap.cc:654] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:09.276219 20531 tablet_bootstrap.cc:654] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:09.277127 20530 tablet_bootstrap.cc:492] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22: No bootstrap required, opened a new log
I20260928 02:59:09.277274 20530 ts_tablet_manager.cc:1397] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:59:09.278420 20530 raft_consensus.cc:348] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } }
I20260928 02:59:09.278720 20530 raft_consensus.cc:374] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:09.278820 20530 raft_consensus.cc:729] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29a5bb2b0074c3fbd25a6be50134e22, State: Initialized, Role: FOLLOWER
I20260928 02:59:09.279093 20530 consensus_queue.cc:260] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } }
I20260928 02:59:09.279893 20530 ts_tablet_manager.cc:1428] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:59:09.284219 20531 tablet_bootstrap.cc:492] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385: No bootstrap required, opened a new log
I20260928 02:59:09.284360 20531 ts_tablet_manager.cc:1397] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260928 02:59:09.284772 20534 tablet_bootstrap.cc:492] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9: Bootstrap starting.
I20260928 02:59:09.285123 20531 raft_consensus.cc:348] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } }
I20260928 02:59:09.285333 20531 raft_consensus.cc:374] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:09.285427 20531 raft_consensus.cc:729] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad760c46b31745509ad8dc6f86e8f385, State: Initialized, Role: FOLLOWER
I20260928 02:59:09.285718 20531 consensus_queue.cc:260] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } }
I20260928 02:59:09.286522 20534 tablet_bootstrap.cc:654] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:09.286708 20297 heartbeater.cc:503] Master 127.12.182.124:40133 was elected leader, sending a full tablet report...
I20260928 02:59:09.290669 20531 ts_tablet_manager.cc:1428] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 02:59:09.292841 20150 heartbeater.cc:503] Master 127.12.182.124:40133 was elected leader, sending a full tablet report...
I20260928 02:59:09.292987 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.280186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.66:33959)
0928 02:59:09.292556 (+ 12370us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.292733 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12222,"threads_started":1}
I20260928 02:59:09.294873 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.293768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.66:52559)
0928 02:59:09.293968 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.294727 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 02:59:09.297554 20534 tablet_bootstrap.cc:492] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9: No bootstrap required, opened a new log
I20260928 02:59:09.297719 20534 ts_tablet_manager.cc:1397] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260928 02:59:09.298353 20534 raft_consensus.cc:348] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } }
I20260928 02:59:09.298521 20534 raft_consensus.cc:374] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:09.298604 20534 raft_consensus.cc:729] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a5462cbbc34ec389f82d89abfe96c9, State: Initialized, Role: FOLLOWER
I20260928 02:59:09.298813 20534 consensus_queue.cc:260] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } }
I20260928 02:59:09.299480 20534 ts_tablet_manager.cc:1428] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:59:09.300307 20537 raft_consensus.cc:482] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:59:09.300508 20537 raft_consensus.cc:504] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } }
I20260928 02:59:09.301323 20438 heartbeater.cc:503] Master 127.12.182.124:40133 was elected leader, sending a full tablet report...
I20260928 02:59:09.302346 20537 leader_election.cc:288] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f29a5bb2b0074c3fbd25a6be50134e22 (127.12.182.66:34761), 86a5462cbbc34ec389f82d89abfe96c9 (127.12.182.67:42661)
I20260928 02:59:09.304083 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.290751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.65:53535)
0928 02:59:09.303585 (+ 12834us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.303788 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12651,"threads_started":1}
I20260928 02:59:09.306569 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.300157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.67:54649)
0928 02:59:09.306166 (+ 6009us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.306313 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5879,"thread_start_us":5768,"threads_started":1}
I20260928 02:59:09.308563 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.307104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.67:45361)
0928 02:59:09.308335 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.308429 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1133}
I20260928 02:59:09.310349 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.303941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:43863 (local address 127.12.182.66:34761)
0928 02:59:09.304095 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.304101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.304117 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:59:09.304173 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.304177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.304243 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.304328 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:09.304944 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.306062 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.308127 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.309275 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.309418 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.309488 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.310000 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.310130 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:59:09.310523 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.302994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:34761 (local address 127.12.182.65:43863)
0928 02:59:09.303685 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.303706 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:09.303831 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:09.304349 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:09.304354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:09.304380 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:09.304770 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.304784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.306233 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.306238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.307983 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.307995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.309501 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.309508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.309766 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.309806 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:59:09.309952 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 02:59:09.310084 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":1,"thread_start_us":2655,"threads_started":1}
I20260928 02:59:09.310870 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.305756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.65:35469)
0928 02:59:09.310357 (+ 4601us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.310601 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4164,"thread_start_us":4017,"threads_started":1}
I20260928 02:59:09.311792 20251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68d0e54622b42f0b7c645f3a142156c" candidate_uuid: "ad760c46b31745509ad8dc6f86e8f385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" is_pre_election: true
I20260928 02:59:09.312145 20251 raft_consensus.cc:2454] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad760c46b31745509ad8dc6f86e8f385 in term 0.
I20260928 02:59:09.312714 20037 leader_election.cc:302] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad760c46b31745509ad8dc6f86e8f385, f29a5bb2b0074c3fbd25a6be50134e22; no voters:
I20260928 02:59:09.313098 20537 raft_consensus.cc:2781] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:59:09.313215 20537 raft_consensus.cc:482] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:59:09.313252 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.306144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:53365 (local address 127.12.182.67:42661)
0928 02:59:09.306308 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.306314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.306330 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:59:09.306388 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.306392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.306456 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.306564 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:09.307007 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.308120 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.310397 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.311371 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.311461 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.311507 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.312918 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.313040 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:59:09.313356 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.304902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:42661 (local address 127.12.182.65:53365)
0928 02:59:09.305028 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.305044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:09.305138 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:09.306581 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:09.306585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:09.306611 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:09.306868 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.306877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.308232 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.308237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.310232 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:09.310246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.312651 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.312655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:09.312774 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.312796 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:59:09.312874 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:59:09.313007 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":23}
I20260928 02:59:09.313325 20537 raft_consensus.cc:3037] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:09.313917 20392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68d0e54622b42f0b7c645f3a142156c" candidate_uuid: "ad760c46b31745509ad8dc6f86e8f385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a5462cbbc34ec389f82d89abfe96c9" is_pre_election: true
I20260928 02:59:09.314105 20392 raft_consensus.cc:2454] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad760c46b31745509ad8dc6f86e8f385 in term 0.
I20260928 02:59:09.315709 20537 raft_consensus.cc:504] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } }
I20260928 02:59:09.316218 20537 leader_election.cc:288] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [CANDIDATE]: Term 1 election: Requested vote from peers f29a5bb2b0074c3fbd25a6be50134e22 (127.12.182.66:34761), 86a5462cbbc34ec389f82d89abfe96c9 (127.12.182.67:42661)
I20260928 02:59:09.316601 20251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68d0e54622b42f0b7c645f3a142156c" candidate_uuid: "ad760c46b31745509ad8dc6f86e8f385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29a5bb2b0074c3fbd25a6be50134e22"
I20260928 02:59:09.316761 20251 raft_consensus.cc:3037] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:09.317799 20392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68d0e54622b42f0b7c645f3a142156c" candidate_uuid: "ad760c46b31745509ad8dc6f86e8f385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a5462cbbc34ec389f82d89abfe96c9"
I20260928 02:59:09.317978 20392 raft_consensus.cc:3037] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:09.319450 20251 raft_consensus.cc:2454] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad760c46b31745509ad8dc6f86e8f385 in term 1.
I20260928 02:59:09.319804 20037 leader_election.cc:302] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad760c46b31745509ad8dc6f86e8f385, f29a5bb2b0074c3fbd25a6be50134e22; no voters:
I20260928 02:59:09.320050 20537 raft_consensus.cc:2781] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:09.320465 20392 raft_consensus.cc:2454] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad760c46b31745509ad8dc6f86e8f385 in term 1.
I20260928 02:59:09.320479 20537 raft_consensus.cc:686] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 1 LEADER]: Becoming Leader. State: Replica: ad760c46b31745509ad8dc6f86e8f385, State: Running, Role: LEADER
I20260928 02:59:09.320855 20537 consensus_queue.cc:237] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } } peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } }
I20260928 02:59:09.321398 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.321071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.65:52747)
0928 02:59:09.321210 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.321287 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:59:09.324116 19970 catalog_manager.cc:5495] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad760c46b31745509ad8dc6f86e8f385 (127.12.182.65). New cstate: current_term: 1 leader_uuid: "ad760c46b31745509ad8dc6f86e8f385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad760c46b31745509ad8dc6f86e8f385" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 45413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 } health_report { overall_health: UNKNOWN } } }
I20260928 02:59:09.343519 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.338627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.12.182.65:45413)
0928 02:59:09.338822 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.338828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.338846 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:59:09.339006 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.339011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.339086 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.339202 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:09.339565 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.340245 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.341490 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.341960 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.342066 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.342450 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:09.342964 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:09.343096 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.343190 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.343265 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:59:09.347150 20392 raft_consensus.cc:1261] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9 [term 1 FOLLOWER]: Refusing update from remote peer ad760c46b31745509ad8dc6f86e8f385: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:09.347548 20251 raft_consensus.cc:1261] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22 [term 1 FOLLOWER]: Refusing update from remote peer ad760c46b31745509ad8dc6f86e8f385: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:09.347841 20544 consensus_queue.cc:1035] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a5462cbbc34ec389f82d89abfe96c9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:09.348227 20537 consensus_queue.cc:1035] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29a5bb2b0074c3fbd25a6be50134e22" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:09.350723 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.350224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.66:55583)
0928 02:59:09.350402 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.350495 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:59:09.351092 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.350644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.67:49597)
0928 02:59:09.350777 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.350912 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:59:09.355438 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.355001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.67:53525)
0928 02:59:09.355210 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.355299 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 02:59:09.357120 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.356693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.66:45869)
0928 02:59:09.356889 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.356970 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 02:59:09.366684 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.366156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.65:56249)
0928 02:59:09.366421 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.366522 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260928 02:59:09.368608 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.368127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.12.182.65:51529)
0928 02:59:09.368362 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.368442 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 02:59:09.373978 19970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:09.374215 19970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:09.378948 19970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2b6b0674db54a908303ea35bce9f8a2: Sending DeleteTablet for 3 replicas of tablet b68d0e54622b42f0b7c645f3a142156c
I20260928 02:59:09.379866 20231 tablet_service.cc:1514] Processing DeleteTablet for tablet b68d0e54622b42f0b7c645f3a142156c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:09 UTC) from {username='oryx', principal='oryx/127.12.182.124@KRBTEST.COM'} at 127.0.0.1:58880
I20260928 02:59:09.380527 20372 tablet_service.cc:1514] Processing DeleteTablet for tablet b68d0e54622b42f0b7c645f3a142156c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:09 UTC) from {username='oryx', principal='oryx/127.12.182.124@KRBTEST.COM'} at 127.0.0.1:59740
I20260928 02:59:09.380781 20558 tablet_replica.cc:330] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22: stopping tablet replica
I20260928 02:59:09.381012 20558 raft_consensus.cc:2229] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:09.381120 20559 tablet_replica.cc:330] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9: stopping tablet replica
I20260928 02:59:09.381289 20558 raft_consensus.cc:2258] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:09.381280 20559 raft_consensus.cc:2229] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:09.381775 20559 raft_consensus.cc:2258] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:09.381932 20084 tablet_service.cc:1514] Processing DeleteTablet for tablet b68d0e54622b42f0b7c645f3a142156c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:09 UTC) from {username='oryx', principal='oryx/127.12.182.124@KRBTEST.COM'} at 127.0.0.1:38164
I20260928 02:59:09.382519 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 19942
I20260928 02:59:09.382556 20558 ts_tablet_manager.cc:1905] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:09.382790 20559 ts_tablet_manager.cc:1905] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:09.388245 20558 ts_tablet_manager.cc:1918] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:09.388389 20558 log.cc:1196] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/b68d0e54622b42f0b7c645f3a142156c
I20260928 02:59:09.388823 20558 ts_tablet_manager.cc:1939] T b68d0e54622b42f0b7c645f3a142156c P f29a5bb2b0074c3fbd25a6be50134e22: Deleting consensus metadata
I20260928 02:59:09.389890 19954 catalog_manager.cc:4841] TS f29a5bb2b0074c3fbd25a6be50134e22 (127.12.182.66:34761): tablet b68d0e54622b42f0b7c645f3a142156c (table test-table [id=9bb4df3ef46e4e01980680e186465d6c]) successfully deleted
I20260928 02:59:09.398743 20560 tablet_replica.cc:330] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385: stopping tablet replica
I20260928 02:59:09.398916 20560 raft_consensus.cc:2229] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:59:09.401247 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.384630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.12.182.65:45413)
0928 02:59:09.384892 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.384898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.384916 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:59:09.384965 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.384970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.385059 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.385149 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:09.386850 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.387948 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.389815 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.390463 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.390563 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.394004 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:09.394018 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:09.394035 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:09.394073 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:09.398064 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:09.399050 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:09.399058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:09.399061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:09.399227 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:09.399642 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:09.399647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:09.399651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:09.400079 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:09.400357 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.400675 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.400872 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260928 02:59:09.401921 20144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:09.402660 20559 ts_tablet_manager.cc:1918] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:09.402791 20559 log.cc:1196] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/b68d0e54622b42f0b7c645f3a142156c
I20260928 02:59:09.403239 20559 ts_tablet_manager.cc:1939] T b68d0e54622b42f0b7c645f3a142156c P 86a5462cbbc34ec389f82d89abfe96c9: Deleting consensus metadata
I20260928 02:59:09.404369 19957 catalog_manager.cc:4841] TS 86a5462cbbc34ec389f82d89abfe96c9 (127.12.182.67:42661): tablet b68d0e54622b42f0b7c645f3a142156c (table test-table [id=9bb4df3ef46e4e01980680e186465d6c]) successfully deleted
I20260928 02:59:09.500038 20560 raft_consensus.cc:2258] T b68d0e54622b42f0b7c645f3a142156c P ad760c46b31745509ad8dc6f86e8f385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:09.500792 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 19942
I20260928 02:59:09.515244 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 20160
I20260928 02:59:09.515878 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.515430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.0.0.1:38184)
0928 02:59:09.515627 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.515711 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:59:09.516283 19957 catalog_manager.cc:4584] TS ad760c46b31745509ad8dc6f86e8f385 (127.12.182.65:45413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b68d0e54622b42f0b7c645f3a142156c: Network error: Client connection negotiation failed: client connection to 127.12.182.65:45413: connect: Connection refused (error 111)
I20260928 02:59:09.527316 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.515985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.12.182.66:34761)
0928 02:59:09.516150 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.516156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.516176 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:59:09.516304 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.516310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.516391 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.516505 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:09.517803 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.518509 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.519611 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.519975 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.520057 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.522379 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:09.522393 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:09.522409 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:09.522440 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:09.525050 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:09.525636 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:09.525645 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:09.525647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:09.525763 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:09.526035 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:09.526038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:09.526040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:09.526350 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:09.526657 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.526828 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.526986 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:59:09.527930 20291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:09.577169 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.576618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.0.0.1:38200)
0928 02:59:09.576871 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.576989 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 02:59:09.578191 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.577819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.0.0.1:38208)
0928 02:59:09.577957 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.578041 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:59:09.643587 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 20160
I20260928 02:59:09.646703 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.646275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.0.0.1:38216)
0928 02:59:09.646473 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.646559 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:59:09.647717 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.647294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.0.0.1:38232)
0928 02:59:09.647458 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.647536 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:59:09.662185 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 20309
I20260928 02:59:09.675243 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.662979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.12.182.67:42661)
0928 02:59:09.663138 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:09.663143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:09.663161 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:59:09.663482 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:09.663487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:09.663571 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:09.663664 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:09.665162 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.666175 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.667446 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:09.667835 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:09.667939 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:09.670374 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:09.670388 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:09.670403 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:09.670445 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:09.673049 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:09.673673 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:09.673679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:09.673681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:09.673796 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:09.674071 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:09.674073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:09.674075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:09.674399 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:09.674615 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:09.674812 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:59:09.674989 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:59:09.675899 20432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:09.749280 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.748827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.0.0.1:38236)
0928 02:59:09.749017 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.749143 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:59:09.750213 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.749908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.0.0.1:38248)
0928 02:59:09.750034 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.750100 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 02:59:09.766811 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 20309
I20260928 02:59:09.780373 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 19645
I20260928 02:59:09.781303 19812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:09.807323 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.806844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50832)
0928 02:59:09.807034 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.807137 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:59:09.808235 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:09.807892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33977 (local address 127.0.0.1:50838)
0928 02:59:09.808020 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:09.808101 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:33977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:59:09.894506 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 19645
I20260928 02:59:09.910566 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 19746
I20260928 02:59:09.911520 20000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:10.037290 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:10.036781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:45413 (local address 127.0.0.1:38258)
0928 02:59:10.036979 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:10.037076 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 02:59:10.037359 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 19746
2026-09-28T02:59:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8671 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:59:10 dist-test-slave-k5rg krb5kdc[20569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:59:10 dist-test-slave-k5rg krb5kdc[20569](info): set up 2 sockets
Sep 28 02:59:10 dist-test-slave-k5rg krb5kdc[20569](info): commencing operation
krb5kdc: starting...
W20260928 02:59:11.038362 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:59:11 dist-test-slave-k5rg krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:59:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.126.keytab.
Entry for principal oryx/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.126.keytab.
I20260928 02:59:11.224231 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:40231
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.126.keytab
--principal=oryx/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:42507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:40231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:11.526350 20582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:11.526623 20582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:11.526747 20582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:11.533842 20582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:59:11.533941 20582 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:59:11.533988 20582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:11.534073 20582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:11.534128 20582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:59:11.534207 20582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:59:11.543856 20582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:40231
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.126.keytab
--principal=oryx/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:40231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:11.545037 20582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:11.546272 20582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:11.557655 20587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:11.557595 20588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:11.557693 20590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:11.563237 20582 server_base.cc:1034] running on GCE node
I20260928 02:59:11.564069 20582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:11.565501 20582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:11.566721 20582 hybrid_clock.cc:648] HybridClock initialized: now 1790564351566658 us; error 43 us; skew 500 ppm
I20260928 02:59:11.567030 20582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:11 dist-test-slave-k5rg krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564351, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:11.569986 20582 init.cc:377] Logged in from keytab as oryx/127.12.182.126@KRBTEST.COM (short username oryx)
I20260928 02:59:11.571420 20582 webserver.cc:460] Webserver started at http://127.12.182.126:45857/ using document root <none> and password file <none>
I20260928 02:59:11.572089 20582 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:11.572171 20582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:11.572423 20582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:11.575228 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "2ec13e6b4f864225862eb941b63972d7"
format_stamp: "Formatted at 2026-09-28 02:59:11 on dist-test-slave-k5rg"
server_key: "a3e5134cb87567145acd6b0d25a07e07"
server_key_iv: "d68313c32eba85d79d8608604479735a"
server_key_version: "encryptionkey@0"
I20260928 02:59:11.575929 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "2ec13e6b4f864225862eb941b63972d7"
format_stamp: "Formatted at 2026-09-28 02:59:11 on dist-test-slave-k5rg"
server_key: "a3e5134cb87567145acd6b0d25a07e07"
server_key_iv: "d68313c32eba85d79d8608604479735a"
server_key_version: "encryptionkey@0"
I20260928 02:59:11.580590 20582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:59:11.583916 20597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:11.585163 20582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:59:11.585295 20582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "2ec13e6b4f864225862eb941b63972d7"
format_stamp: "Formatted at 2026-09-28 02:59:11 on dist-test-slave-k5rg"
server_key: "a3e5134cb87567145acd6b0d25a07e07"
server_key_iv: "d68313c32eba85d79d8608604479735a"
server_key_version: "encryptionkey@0"
I20260928 02:59:11.585456 20582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:11.600224 20582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:11.603920 20582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:11.604130 20582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:11.623586 20582 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:40231
I20260928 02:59:11.623605 20658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:40231 every 8 connection(s)
I20260928 02:59:11.624806 20582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:59:11.628217 20659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:11.633149 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 20582
I20260928 02:59:11.633313 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:59:11.633996 13017 external_mini_cluster.cc:1428] Setting key 89cf3966925f4d3e70e741270f8a542d
I20260928 02:59:11.638080 20659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7: Bootstrap starting.
I20260928 02:59:11.641337 20659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:11.642391 20659 log.cc:824] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:11.645125 20659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7: No bootstrap required, opened a new log
Sep 28 02:59:11 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:11.652781 20659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec13e6b4f864225862eb941b63972d7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40231 } }
I20260928 02:59:11.653113 20659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:11.653215 20659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec13e6b4f864225862eb941b63972d7, State: Initialized, Role: FOLLOWER
I20260928 02:59:11.653841 20659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec13e6b4f864225862eb941b63972d7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40231 } }
I20260928 02:59:11.654099 20659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:59:11.654237 20659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:59:11.654417 20659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:11.655575 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:11.635824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.12.182.126:40231)
0928 02:59:11.636625 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:11.636636 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:11.636673 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:59:11.638314 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:11.638335 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:11.638679 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:11.638959 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:11.640622 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:11.641935 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:11.643425 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:11.644106 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:11.644229 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:11.648607 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:11.648614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:11.648631 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:11.648666 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:11.651466 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:11.652088 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:11.652094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:11.652100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:11.652254 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:11.652612 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:11.652615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:11.652617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:11.653148 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:11.653414 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:11.653861 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 02:59:11.654332 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":377,"threads_started":1}
I20260928 02:59:11.656971 20659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec13e6b4f864225862eb941b63972d7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40231 } }
I20260928 02:59:11.657415 20659 leader_election.cc:302] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ec13e6b4f864225862eb941b63972d7; no voters:
I20260928 02:59:11.657994 20659 leader_election.cc:288] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:59:11.658195 20664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:11.658839 20664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [term 1 LEADER]: Becoming Leader. State: Replica: 2ec13e6b4f864225862eb941b63972d7, State: Running, Role: LEADER
I20260928 02:59:11.659291 20664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec13e6b4f864225862eb941b63972d7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40231 } }
I20260928 02:59:11.659765 20659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:59:11.661746 20666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec13e6b4f864225862eb941b63972d7. Latest consensus state: current_term: 1 leader_uuid: "2ec13e6b4f864225862eb941b63972d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec13e6b4f864225862eb941b63972d7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40231 } } }
I20260928 02:59:11.662075 20666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:11.662882 20665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ec13e6b4f864225862eb941b63972d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec13e6b4f864225862eb941b63972d7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40231 } } }
I20260928 02:59:11.663244 20665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:11.668561 20669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:59:11.673343 20669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:59:11.680974 20669 catalog_manager.cc:1347] Generated new cluster ID: a5cddefa113b4429bdb8ffbb29b78561
I20260928 02:59:11.681071 20669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:59:11.705643 20669 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:59:11.706944 20669 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:59:11.720407 20669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7: Generated new TSK 0
I20260928 02:59:11.721385 20669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab.
Entry for principal oryx/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab.
I20260928 02:59:11.769119 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40231
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab
--principal=oryx/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:42507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:12.070871 20685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:12.071180 20685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:12.071316 20685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:12.078521 20685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:12.078634 20685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:12.078755 20685 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:59:12.078862 20685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:59:12.088619 20685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab
--principal=oryx/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.182.126:40231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:12.090040 20685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:12.091336 20685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:12.103430 20691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:12.103384 20693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:12.103451 20690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:12.104323 20685 server_base.cc:1034] running on GCE node
I20260928 02:59:12.104874 20685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:12.105770 20685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:12.106971 20685 hybrid_clock.cc:648] HybridClock initialized: now 1790564352106911 us; error 41 us; skew 500 ppm
I20260928 02:59:12.107244 20685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:12 dist-test-slave-k5rg krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564352, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:12.110347 20685 init.cc:377] Logged in from keytab as oryx/127.12.182.65@KRBTEST.COM (short username oryx)
I20260928 02:59:12.111650 20685 webserver.cc:460] Webserver started at http://127.12.182.65:34571/ using document root <none> and password file <none>
I20260928 02:59:12.112257 20685 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:12.112339 20685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:12.112591 20685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:12.115306 20685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "01e74da57e85432a890c2241dd3ef877"
format_stamp: "Formatted at 2026-09-28 02:59:12 on dist-test-slave-k5rg"
server_key: "e8b76fa3c089faf566358f864a40afb1"
server_key_iv: "6c63fbfca86e91cf41d02c85ce9457e7"
server_key_version: "encryptionkey@0"
I20260928 02:59:12.116003 20685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "01e74da57e85432a890c2241dd3ef877"
format_stamp: "Formatted at 2026-09-28 02:59:12 on dist-test-slave-k5rg"
server_key: "e8b76fa3c089faf566358f864a40afb1"
server_key_iv: "6c63fbfca86e91cf41d02c85ce9457e7"
server_key_version: "encryptionkey@0"
I20260928 02:59:12.120553 20685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:59:12.123708 20700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:12.124897 20685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:59:12.125027 20685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "01e74da57e85432a890c2241dd3ef877"
format_stamp: "Formatted at 2026-09-28 02:59:12 on dist-test-slave-k5rg"
server_key: "e8b76fa3c089faf566358f864a40afb1"
server_key_iv: "6c63fbfca86e91cf41d02c85ce9457e7"
server_key_version: "encryptionkey@0"
I20260928 02:59:12.125190 20685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:12.138777 20685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:12.142624 20685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:12.142827 20685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:12.145161 20685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:12.145234 20685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:12.145347 20685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:12.145403 20685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:59:12 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564352, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:12.170014 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:12.146322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40231 (local address 127.12.182.65:49517)
0928 02:59:12.147187 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:12.147235 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:12.149477 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:12.150357 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:12.150372 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:12.151193 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:12.152149 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:12.152181 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.153583 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.153588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:12.155796 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:12.155814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.156813 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.156821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:12.157118 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:12.157936 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:12.157969 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:12.161254 (+ 3285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:12.165037 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:12.165046 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:12.165062 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:12.165644 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:12.166304 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:12.166309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:12.166312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:12.166620 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:12.167282 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:12.167288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:12.167737 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 02:59:12.168043 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 02:59:12.168571 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":337,"threads_started":1}
I20260928 02:59:12.171054 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:12.145814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:49517 (local address 127.12.182.126:40231)
0928 02:59:12.145991 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:12.145998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:12.147370 (+ 1372us) server_negotiation.cc:407] Connection header received
0928 02:59:12.149874 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:12.149880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:12.149968 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:12.150071 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:12.152354 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.153361 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.155943 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.156560 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.156692 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:12.161536 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:12.161543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:12.161548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:12.161595 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:12.164880 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:12.165763 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:12.165768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:12.165771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:12.165948 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:12.166729 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:12.166734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:12.166737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:12.167074 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:12.167214 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:12.170506 (+ 3292us) server_negotiation.cc:299] Negotiation successful
0928 02:59:12.170727 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:59:12.188170 20685 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:40191
I20260928 02:59:12.188201 20819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:40191 every 8 connection(s)
I20260928 02:59:12.189633 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:12.176423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:39581 (local address 127.12.182.126:40231)
0928 02:59:12.176594 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:12.176600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:12.177441 (+ 841us) server_negotiation.cc:407] Connection header received
0928 02:59:12.177598 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:12.177621 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:12.177705 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:12.177807 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:12.178638 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.179685 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.181426 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.182076 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.182203 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:12.183991 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:12.183998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:12.184004 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:12.184042 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:12.186894 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:12.187742 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:12.187748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:12.187751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:12.187903 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:12.188307 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:12.188312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:12.188316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:12.188646 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:12.188761 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:12.189199 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 02:59:12.189365 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:59:12.189733 20685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:59:12.189963 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:12.176682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40231 (local address 127.12.182.65:39581)
0928 02:59:12.177314 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:12.177333 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:12.177482 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:12.177851 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:12.177859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:12.178238 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:12.178511 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:12.178520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.179817 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.179823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:12.181276 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:12.181291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.182280 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.182392 (+ 112us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:12.182541 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:12.183140 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:12.183157 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:12.183804 (+ 647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:12.187012 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:12.187018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:12.187023 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:12.187614 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:12.188022 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:12.188027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:12.188031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:12.188194 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:12.188763 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:12.188767 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:12.188965 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 02:59:12.189139 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:59:12.189460 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":321,"threads_started":1}
I20260928 02:59:12.191771 20820 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40231
I20260928 02:59:12.192003 20820 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:12.192776 20820 heartbeater.cc:511] Master 127.12.182.126:40231 requested a full tablet report, sending...
I20260928 02:59:12.194581 20614 ts_manager.cc:194] Registered new tserver with Master: 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191)
I20260928 02:59:12.195889 20614 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:49517
I20260928 02:59:12.197296 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 20685
I20260928 02:59:12.197468 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:59:12.198098 13017 external_mini_cluster.cc:1428] Setting key c29d4589eaa3d0df4c1fa5ac606a859b
WARNING: no policy specified for oryx/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab.
Entry for principal oryx/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab.
I20260928 02:59:12.238374 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40231
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab
--principal=oryx/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:42507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:12.539538 20824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:12.539786 20824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:12.539925 20824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:12.546931 20824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:12.547027 20824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:12.547147 20824 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:59:12.547250 20824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:59:12.556438 20824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab
--principal=oryx/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.182.126:40231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:12.557617 20824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:12.558836 20824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:12.571023 20830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:12.571054 20832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:12.571096 20829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:12.571817 20824 server_base.cc:1034] running on GCE node
I20260928 02:59:12.572398 20824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:12.573274 20824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:12.574477 20824 hybrid_clock.cc:648] HybridClock initialized: now 1790564352574412 us; error 42 us; skew 500 ppm
I20260928 02:59:12.574743 20824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:12 dist-test-slave-k5rg krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564352, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:12.577549 20824 init.cc:377] Logged in from keytab as oryx/127.12.182.66@KRBTEST.COM (short username oryx)
I20260928 02:59:12.578886 20824 webserver.cc:460] Webserver started at http://127.12.182.66:36043/ using document root <none> and password file <none>
I20260928 02:59:12.579491 20824 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:12.579573 20824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:12.579821 20824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:12.582551 20824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "d9cd2557112c43a68222d2fec41cce57"
format_stamp: "Formatted at 2026-09-28 02:59:12 on dist-test-slave-k5rg"
server_key: "3bc1ca79e040ed7a3e252cc57684e9fe"
server_key_iv: "f3584845cbcdce676da6dbadeb72b76f"
server_key_version: "encryptionkey@0"
I20260928 02:59:12.583290 20824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "d9cd2557112c43a68222d2fec41cce57"
format_stamp: "Formatted at 2026-09-28 02:59:12 on dist-test-slave-k5rg"
server_key: "3bc1ca79e040ed7a3e252cc57684e9fe"
server_key_iv: "f3584845cbcdce676da6dbadeb72b76f"
server_key_version: "encryptionkey@0"
I20260928 02:59:12.588140 20824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:59:12.591332 20839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:12.592521 20824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:59:12.592636 20824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "d9cd2557112c43a68222d2fec41cce57"
format_stamp: "Formatted at 2026-09-28 02:59:12 on dist-test-slave-k5rg"
server_key: "3bc1ca79e040ed7a3e252cc57684e9fe"
server_key_iv: "f3584845cbcdce676da6dbadeb72b76f"
server_key_version: "encryptionkey@0"
I20260928 02:59:12.592806 20824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:12.612228 20824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:12.615906 20824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:12.616114 20824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:12.618665 20824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:12.618768 20824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:12.618888 20824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:12.618937 20824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:59:12 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564352, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:12.643383 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:12.618944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:36685 (local address 127.12.182.126:40231)
0928 02:59:12.619152 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:12.619159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:12.620482 (+ 1323us) server_negotiation.cc:407] Connection header received
0928 02:59:12.623150 (+ 2668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:12.623156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:12.623243 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:12.623323 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:12.625759 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.626964 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.629802 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.630504 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.630600 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:12.635359 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:12.635365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:12.635370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:12.635419 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:12.638696 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:12.639634 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:12.639639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:12.639646 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:12.639774 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:12.640278 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:12.640283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:12.640286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:12.640623 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:12.640755 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:12.642783 (+ 2028us) server_negotiation.cc:299] Negotiation successful
0928 02:59:12.643049 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:59:12.643543 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:12.619517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40231 (local address 127.12.182.66:36685)
0928 02:59:12.620315 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:12.620364 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:12.622820 (+ 2456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:12.623829 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:12.623865 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:12.624626 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:12.625517 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:12.625544 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.627195 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.627200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:12.629626 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:12.629641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.630876 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.630884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:12.631168 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:12.631898 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:12.631935 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:12.635114 (+ 3179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:12.638853 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:12.638862 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:12.638878 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:12.639481 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:12.639906 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:12.639910 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:12.639914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:12.640156 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:12.640826 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:12.640834 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:12.641249 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 02:59:12.641542 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 02:59:12.642089 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":322,"threads_started":1}
I20260928 02:59:12.656970 20824 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:35963
I20260928 02:59:12.657050 20958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:35963 every 8 connection(s)
I20260928 02:59:12.658384 20824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:59:12.660204 20959 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40231
I20260928 02:59:12.660403 20959 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:12.661000 20959 heartbeater.cc:511] Master 127.12.182.126:40231 requested a full tablet report, sending...
I20260928 02:59:12.662228 20614 ts_manager.cc:194] Registered new tserver with Master: d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963)
I20260928 02:59:12.663009 20614 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:36685
I20260928 02:59:12.663481 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:12.649784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:35425 (local address 127.12.182.126:40231)
0928 02:59:12.649922 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:12.649926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:12.650700 (+ 774us) server_negotiation.cc:407] Connection header received
0928 02:59:12.650864 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:12.650869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:12.650935 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:12.651025 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:12.652457 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.653380 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.655463 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.655905 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.656024 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:12.658146 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:12.658151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:12.658154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:12.658182 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:12.661048 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:12.661664 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:12.661669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:12.661672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:12.661778 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:12.662095 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:12.662099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:12.662103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:12.662431 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:12.662553 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:12.662920 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:59:12.663117 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:59:12.663530 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:12.649886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40231 (local address 127.12.182.66:35425)
0928 02:59:12.650614 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:12.650636 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:12.650763 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:12.651065 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:12.651070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:12.651803 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:12.652282 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:12.652297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.653507 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.653512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:12.655336 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:12.655354 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:12.656103 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:12.656110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:12.656298 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:12.657015 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:12.657041 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:12.657992 (+ 951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:12.661157 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:12.661163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:12.661167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:12.661545 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:12.661873 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:12.661878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:12.661882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:12.661997 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:12.662533 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:12.662538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:12.662693 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 02:59:12.662796 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:59:12.663077 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":394,"threads_started":1}
I20260928 02:59:12.664808 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 20824
I20260928 02:59:12.664973 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:59:12.665504 13017 external_mini_cluster.cc:1428] Setting key 11ebe053ca6ac750140f06ef5caec3d4
WARNING: no policy specified for oryx/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab.
Entry for principal oryx/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab.
I20260928 02:59:12.705799 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40231
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab
--principal=oryx/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:42507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:13.012583 20963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:13.012851 20963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:13.012979 20963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:13.020002 20963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:13.020117 20963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:13.020237 20963 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:59:13.020347 20963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:59:13.029646 20963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab
--principal=oryx/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.182.126:40231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:13.030788 20963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:13.032068 20963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:13.044289 20968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:13.044348 20971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:13.044384 20969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:13.044966 20963 server_base.cc:1034] running on GCE node
I20260928 02:59:13.045548 20963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:13.046407 20963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:13.047642 20963 hybrid_clock.cc:648] HybridClock initialized: now 1790564353047580 us; error 28 us; skew 500 ppm
I20260928 02:59:13.047982 20963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:13 dist-test-slave-k5rg krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564353, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:13.054761 20963 init.cc:377] Logged in from keytab as oryx/127.12.182.67@KRBTEST.COM (short username oryx)
I20260928 02:59:13.056135 20963 webserver.cc:460] Webserver started at http://127.12.182.67:44673/ using document root <none> and password file <none>
I20260928 02:59:13.056771 20963 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:13.056866 20963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:13.057165 20963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:13.059918 20963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "a5c3143914ee47a5af2e53283e380f91"
format_stamp: "Formatted at 2026-09-28 02:59:13 on dist-test-slave-k5rg"
server_key: "488472f81cd430b5627ca661dbfccccd"
server_key_iv: "d818f53d8e19ae695c0f41bfd3024770"
server_key_version: "encryptionkey@0"
I20260928 02:59:13.060678 20963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "a5c3143914ee47a5af2e53283e380f91"
format_stamp: "Formatted at 2026-09-28 02:59:13 on dist-test-slave-k5rg"
server_key: "488472f81cd430b5627ca661dbfccccd"
server_key_iv: "d818f53d8e19ae695c0f41bfd3024770"
server_key_version: "encryptionkey@0"
I20260928 02:59:13.065512 20963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:59:13.068938 20978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:13.070261 20963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:59:13.070400 20963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "a5c3143914ee47a5af2e53283e380f91"
format_stamp: "Formatted at 2026-09-28 02:59:13 on dist-test-slave-k5rg"
server_key: "488472f81cd430b5627ca661dbfccccd"
server_key_iv: "d818f53d8e19ae695c0f41bfd3024770"
server_key_version: "encryptionkey@0"
I20260928 02:59:13.070608 20963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:13.116994 20963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:13.120955 20963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:13.121236 20963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:13.123760 20963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:13.123855 20963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:13.123982 20963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:13.124048 20963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:59:13 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564353, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:13.146070 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.124602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:52853 (local address 127.12.182.126:40231)
0928 02:59:13.124786 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.124792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.126126 (+ 1334us) server_negotiation.cc:407] Connection header received
0928 02:59:13.128144 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.128150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.128233 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.128318 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:13.130556 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.131530 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.134066 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.134607 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.134745 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.139167 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:13.139174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:13.139180 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:13.139226 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:13.142612 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:13.143554 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:13.143559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:13.143563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:13.143731 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:13.144299 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:13.144304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:13.144307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:13.144607 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:13.144737 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.145456 (+ 719us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.145713 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:59:13.147024 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.125135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40231 (local address 127.12.182.67:52853)
0928 02:59:13.125958 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.126009 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.127828 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.128619 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.128632 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.129452 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:13.130313 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.130345 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.131709 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.131715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.133854 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.133867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.134883 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.134890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.135205 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.136076 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:13.136109 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:13.138901 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:13.142760 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:13.142769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:13.142791 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:13.143432 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:13.143874 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:13.143880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:13.143884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:13.144162 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:13.144729 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:13.144737 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:13.145107 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.145337 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.145871 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":342,"threads_started":1}
I20260928 02:59:13.166533 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.152933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:43349 (local address 127.12.182.126:40231)
0928 02:59:13.153113 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.153119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.153676 (+ 557us) server_negotiation.cc:407] Connection header received
0928 02:59:13.153823 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.153828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.153908 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.154012 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:13.154924 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.156076 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.157323 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.158052 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.158177 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.160247 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:13.160254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:13.160260 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:13.160300 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:13.163688 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:13.164329 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:13.164334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:13.164338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:13.164513 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:13.164896 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:13.164902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:13.164906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:13.165239 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:13.165386 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.165937 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.166145 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:59:13.166611 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.153029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40231 (local address 127.12.182.67:43349)
0928 02:59:13.153580 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.153616 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.153726 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.154025 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.154028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.154516 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:13.154801 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.154810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.156200 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.156204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.157199 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.157210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.158207 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.158211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.158324 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.158867 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:13.158933 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:13.159937 (+ 1004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:13.163821 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:13.163826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:13.163831 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:13.164232 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:13.164643 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:13.164647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:13.164649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:13.164769 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:13.165378 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:13.165415 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:13.165676 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.165809 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.166194 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":285,"threads_started":1}
I20260928 02:59:13.175314 20963 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:41541
I20260928 02:59:13.175341 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:41541 every 8 connection(s)
I20260928 02:59:13.176429 20963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:59:13.178203 21098 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40231
I20260928 02:59:13.178402 21098 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:13.178966 21098 heartbeater.cc:511] Master 127.12.182.126:40231 requested a full tablet report, sending...
I20260928 02:59:13.180222 20614 ts_manager.cc:194] Registered new tserver with Master: a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541)
I20260928 02:59:13.180984 20614 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:52853
I20260928 02:59:13.182340 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 20963
I20260928 02:59:13.182476 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:59:13.182911 13017 external_mini_cluster.cc:1428] Setting key 62ae58d236fe1a9f48568c4bf1d6e6e7
I20260928 02:59:13.185644 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:59:13.198328 20820 heartbeater.cc:503] Master 127.12.182.126:40231 was elected leader, sending a full tablet report...
Sep 28 02:59:13 dist-test-slave-k5rg krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564353, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:59:13 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:13 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:59:13.228971 20662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:13.219559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.12.182.126:40231)
0928 02:59:13.219766 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.219773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.219914 (+ 141us) server_negotiation.cc:407] Connection header received
0928 02:59:13.220103 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.220107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.220175 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.220259 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:13.221540 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.222294 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.223522 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.223941 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.224055 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.228757 (+ 4702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51852: BlockingRecv error: recv got EOF from 127.0.0.1:51852 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 02:59:13 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564353, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:13.243091 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.231722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.12.182.126:40231)
0928 02:59:13.231877 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.231880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.232137 (+ 257us) server_negotiation.cc:407] Connection header received
0928 02:59:13.232253 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.232256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.232310 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.232385 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:13.233357 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.234084 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.235163 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.235545 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.235627 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.238869 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:13.238874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:13.238878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:13.238907 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:13.241053 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:13.241631 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:13.241637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:13.241641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:13.241751 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:13.242021 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:13.242024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:13.242026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:13.242275 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:13.242444 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.242648 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.242796 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:59:13.246743 20614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:59:13.253409 20614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:59:13.290095 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.264891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:41541 (local address 127.0.0.1:44204)
0928 02:59:13.279613 (+ 14722us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.279634 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.279766 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.281428 (+ 1662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.281437 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.281468 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:13.283085 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.283104 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.285139 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.285143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.287353 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.287366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.289025 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.289031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.289271 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.289334 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.289455 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.289668 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14574,"mutex_wait_us":1,"thread_start_us":319,"threads_started":1}
I20260928 02:59:13.290455 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.264319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:35963 (local address 127.0.0.1:60630)
0928 02:59:13.265659 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.265707 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.265891 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.282798 (+ 16907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.282803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.282830 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:13.283461 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.283476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.285801 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.285807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.288167 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.288179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.289493 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.289500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.289728 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.289761 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.289900 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.290027 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":20,"thread_start_us":15529,"threads_started":1}
I20260928 02:59:13.290812 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.264549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.12.182.66:35963)
0928 02:59:13.282146 (+ 17597us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.282154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.282184 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:59:13.282280 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.282288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.282542 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.282724 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:13.284163 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.285678 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.288307 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.289262 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.289381 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.289590 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.289964 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.290130 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17455,"thread_start_us":17356,"threads_started":1}
I20260928 02:59:13.291081 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.265083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.12.182.67:41541)
0928 02:59:13.280748 (+ 15665us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.280759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.280801 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:59:13.280908 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.280916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.281195 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.281412 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:13.283329 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.284992 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.287533 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.288831 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.290321 (+ 1490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.290550 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.290628 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.290768 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15469,"thread_start_us":16613,"threads_started":1}
I20260928 02:59:13.292479 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.266042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40191 (local address 127.0.0.1:58456)
0928 02:59:13.283874 (+ 17832us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.283893 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.284014 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.285325 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.285329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.285354 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:13.285745 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.285757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.287607 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.287612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.289849 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.289863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.291799 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.291805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.291963 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.291992 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.292071 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.292184 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17690,"thread_start_us":17557,"threads_started":1}
I20260928 02:59:13.292562 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.266231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.12.182.65:40191)
0928 02:59:13.281395 (+ 15164us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.281403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.284641 (+ 3238us) server_negotiation.cc:407] Connection header received
0928 02:59:13.284735 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.284742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.284963 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.285126 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:13.285962 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.287453 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.290310 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.291609 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.291720 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.291903 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.292146 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.292307 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15016,"thread_start_us":15174,"threads_started":1}
I20260928 02:59:13.294693 21028 tablet_service.cc:1467] Processing CreateTablet for tablet 720c35c478274da9989ad1811aa0093e (DEFAULT_TABLE table=test-table [id=8e8ab20431a743fe809b1ae5af8104ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:13.294730 20889 tablet_service.cc:1467] Processing CreateTablet for tablet 720c35c478274da9989ad1811aa0093e (DEFAULT_TABLE table=test-table [id=8e8ab20431a743fe809b1ae5af8104ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:13.296041 20889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720c35c478274da9989ad1811aa0093e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:13.296043 21028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720c35c478274da9989ad1811aa0093e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:13.295748 20750 tablet_service.cc:1467] Processing CreateTablet for tablet 720c35c478274da9989ad1811aa0093e (DEFAULT_TABLE table=test-table [id=8e8ab20431a743fe809b1ae5af8104ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:13.296890 20750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720c35c478274da9989ad1811aa0093e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:13.307933 21122 tablet_bootstrap.cc:492] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: Bootstrap starting.
I20260928 02:59:13.310010 21123 tablet_bootstrap.cc:492] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: Bootstrap starting.
I20260928 02:59:13.310699 21124 tablet_bootstrap.cc:492] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: Bootstrap starting.
I20260928 02:59:13.311278 21122 tablet_bootstrap.cc:654] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:13.312469 21122 log.cc:824] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:13.313862 21124 tablet_bootstrap.cc:654] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:13.315047 21124 log.cc:824] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:13.316422 21123 tablet_bootstrap.cc:654] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:13.317324 21124 tablet_bootstrap.cc:492] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: No bootstrap required, opened a new log
I20260928 02:59:13.317381 21122 tablet_bootstrap.cc:492] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: No bootstrap required, opened a new log
I20260928 02:59:13.317556 21124 ts_tablet_manager.cc:1397] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260928 02:59:13.317559 21123 log.cc:824] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:13.317704 21122 ts_tablet_manager.cc:1397] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260928 02:59:13.320291 21123 tablet_bootstrap.cc:492] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: No bootstrap required, opened a new log
I20260928 02:59:13.320538 21123 ts_tablet_manager.cc:1397] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260928 02:59:13.324877 21124 raft_consensus.cc:348] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.325199 21124 raft_consensus.cc:374] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:13.325316 21124 raft_consensus.cc:729] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cd2557112c43a68222d2fec41cce57, State: Initialized, Role: FOLLOWER
I20260928 02:59:13.325902 21124 consensus_queue.cc:260] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.326814 20959 heartbeater.cc:503] Master 127.12.182.126:40231 was elected leader, sending a full tablet report...
I20260928 02:59:13.327278 21122 raft_consensus.cc:348] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.327651 21122 raft_consensus.cc:374] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:13.327787 21122 raft_consensus.cc:729] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c3143914ee47a5af2e53283e380f91, State: Initialized, Role: FOLLOWER
I20260928 02:59:13.328682 21124 ts_tablet_manager.cc:1428] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260928 02:59:13.330446 21123 raft_consensus.cc:348] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.330907 21123 raft_consensus.cc:374] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:13.331034 21123 raft_consensus.cc:729] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e74da57e85432a890c2241dd3ef877, State: Initialized, Role: FOLLOWER
I20260928 02:59:13.331640 21123 consensus_queue.cc:260] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.333215 21123 ts_tablet_manager.cc:1428] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260928 02:59:13.333963 21122 consensus_queue.cc:260] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.334338 21129 raft_consensus.cc:482] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:59:13.334661 21129 raft_consensus.cc:504] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.336520 21129 leader_election.cc:288] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963), 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191)
I20260928 02:59:13.337128 21098 heartbeater.cc:503] Master 127.12.182.126:40231 was elected leader, sending a full tablet report...
I20260928 02:59:13.341997 21122 ts_tablet_manager.cc:1428] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: Time spent starting tablet: real 0.024s user 0.015s sys 0.008s
I20260928 02:59:13.344394 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.338426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:41765 (local address 127.12.182.66:35963)
0928 02:59:13.338607 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.338612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.338632 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:59:13.338690 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.338695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.338760 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.338848 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:13.339445 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.340565 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.342602 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.343462 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.343608 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.343673 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.344113 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.344210 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:59:13.344587 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.336900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:35963 (local address 127.12.182.67:41765)
0928 02:59:13.337085 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.337105 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.337226 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.338999 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.339004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.339030 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:13.339278 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.339289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.340699 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.340704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.342472 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.342482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.343756 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.343762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.343991 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.344015 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.344087 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.344264 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":26}
I20260928 02:59:13.344655 21132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.337304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40191 (local address 127.12.182.67:48441)
0928 02:59:13.337941 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.337955 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.338068 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.338488 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.338492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.338516 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:13.338850 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.338864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.340049 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.340052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.342070 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.342086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.343545 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.343550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.343925 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.343994 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.344087 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.344264 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":88,"thread_start_us":339,"threads_started":1}
I20260928 02:59:13.344983 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.336887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:48441 (local address 127.12.182.65:40191)
0928 02:59:13.337065 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.337070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.338029 (+ 959us) server_negotiation.cc:407] Connection header received
0928 02:59:13.338182 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.338187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.338245 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.338320 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:13.339033 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.339916 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.342217 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.343344 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.343524 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.343593 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.344651 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.344790 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:59:13.345537 20909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720c35c478274da9989ad1811aa0093e" candidate_uuid: "a5c3143914ee47a5af2e53283e380f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9cd2557112c43a68222d2fec41cce57" is_pre_election: true
I20260928 02:59:13.345849 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720c35c478274da9989ad1811aa0093e" candidate_uuid: "a5c3143914ee47a5af2e53283e380f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e74da57e85432a890c2241dd3ef877" is_pre_election: true
I20260928 02:59:13.346110 20909 raft_consensus.cc:2454] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5c3143914ee47a5af2e53283e380f91 in term 0.
I20260928 02:59:13.346268 20770 raft_consensus.cc:2454] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5c3143914ee47a5af2e53283e380f91 in term 0.
I20260928 02:59:13.346832 20980 leader_election.cc:302] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5c3143914ee47a5af2e53283e380f91, d9cd2557112c43a68222d2fec41cce57; no voters:
I20260928 02:59:13.347223 21131 raft_consensus.cc:2781] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:59:13.347354 21131 raft_consensus.cc:482] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:59:13.347514 21131 raft_consensus.cc:3037] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:13.349793 21131 raft_consensus.cc:504] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.350369 21131 leader_election.cc:288] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [CANDIDATE]: Term 1 election: Requested vote from peers d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963), 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191)
I20260928 02:59:13.350728 20909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720c35c478274da9989ad1811aa0093e" candidate_uuid: "a5c3143914ee47a5af2e53283e380f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9cd2557112c43a68222d2fec41cce57"
I20260928 02:59:13.350837 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720c35c478274da9989ad1811aa0093e" candidate_uuid: "a5c3143914ee47a5af2e53283e380f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e74da57e85432a890c2241dd3ef877"
I20260928 02:59:13.350965 20909 raft_consensus.cc:3037] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:13.351007 20770 raft_consensus.cc:3037] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:13.353961 20770 raft_consensus.cc:2454] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5c3143914ee47a5af2e53283e380f91 in term 1.
I20260928 02:59:13.353973 20909 raft_consensus.cc:2454] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5c3143914ee47a5af2e53283e380f91 in term 1.
I20260928 02:59:13.354323 20980 leader_election.cc:302] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01e74da57e85432a890c2241dd3ef877, a5c3143914ee47a5af2e53283e380f91, d9cd2557112c43a68222d2fec41cce57; no voters:
I20260928 02:59:13.354609 21131 raft_consensus.cc:2781] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:13.354774 21131 raft_consensus.cc:686] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 1 LEADER]: Becoming Leader. State: Replica: a5c3143914ee47a5af2e53283e380f91, State: Running, Role: LEADER
I20260928 02:59:13.355175 21131 consensus_queue.cc:237] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.359530 20614 catalog_manager.cc:5495] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5c3143914ee47a5af2e53283e380f91 (127.12.182.67). New cstate: current_term: 1 leader_uuid: "a5c3143914ee47a5af2e53283e380f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } health_report { overall_health: UNKNOWN } } }
I20260928 02:59:13.373498 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.368638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.12.182.67:41541)
0928 02:59:13.368780 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.368784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.368818 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:59:13.369032 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.369035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.369102 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.369212 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:13.369473 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.370406 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.371728 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.372137 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.372224 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.372598 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:13.373067 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:13.373172 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.373256 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.373313 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:59:13.380722 20770 raft_consensus.cc:1261] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Refusing update from remote peer a5c3143914ee47a5af2e53283e380f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:13.380723 20909 raft_consensus.cc:1261] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Refusing update from remote peer a5c3143914ee47a5af2e53283e380f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:13.381731 21131 consensus_queue.cc:1035] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:13.382014 21129 consensus_queue.cc:1035] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:13.401276 21135 mvcc.cc:205] Tried to move back new op lower bound from 7334151591438725120 to 7334151591346696192. Current Snapshot: MvccSnapshot[applied={T|T < 7334151591438725120 or (T in {7334151591438725120})}]
I20260928 02:59:13.414080 20613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:13.414340 20613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:13.417928 20613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7: Sending DeleteTablet for 3 replicas of tablet 720c35c478274da9989ad1811aa0093e
I20260928 02:59:13.418864 20889 tablet_service.cc:1514] Processing DeleteTablet for tablet 720c35c478274da9989ad1811aa0093e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:13 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:60630
I20260928 02:59:13.419096 21028 tablet_service.cc:1514] Processing DeleteTablet for tablet 720c35c478274da9989ad1811aa0093e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:13 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:44204
I20260928 02:59:13.420095 20750 tablet_service.cc:1514] Processing DeleteTablet for tablet 720c35c478274da9989ad1811aa0093e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:13 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:58456
I20260928 02:59:13.425287 20613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:59:13.425978 20613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:59:13.435048 21027 tablet_service.cc:1467] Processing CreateTablet for tablet 55a5ff83dc1a42a0a819a07fcb11097f (DEFAULT_TABLE table=test-table [id=ed8e246d865c409aa1c916887224e354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:13.435046 20888 tablet_service.cc:1467] Processing CreateTablet for tablet 55a5ff83dc1a42a0a819a07fcb11097f (DEFAULT_TABLE table=test-table [id=ed8e246d865c409aa1c916887224e354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:13.435096 20749 tablet_service.cc:1467] Processing CreateTablet for tablet 55a5ff83dc1a42a0a819a07fcb11097f (DEFAULT_TABLE table=test-table [id=ed8e246d865c409aa1c916887224e354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:13.435559 21027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a5ff83dc1a42a0a819a07fcb11097f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:13.435568 20749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a5ff83dc1a42a0a819a07fcb11097f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:13.435647 20888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a5ff83dc1a42a0a819a07fcb11097f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:13.441890 21123 tablet_bootstrap.cc:492] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: Bootstrap starting.
I20260928 02:59:13.442642 21124 tablet_bootstrap.cc:492] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: Bootstrap starting.
I20260928 02:59:13.443562 21123 tablet_bootstrap.cc:654] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:13.444342 21124 tablet_bootstrap.cc:654] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:13.446166 21122 tablet_bootstrap.cc:492] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: Bootstrap starting.
I20260928 02:59:13.447194 21123 tablet_bootstrap.cc:492] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: No bootstrap required, opened a new log
I20260928 02:59:13.447397 21123 ts_tablet_manager.cc:1397] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260928 02:59:13.447690 21122 tablet_bootstrap.cc:654] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:13.448324 21123 raft_consensus.cc:348] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:13.448643 21123 raft_consensus.cc:374] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:13.448770 21123 raft_consensus.cc:729] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e74da57e85432a890c2241dd3ef877, State: Initialized, Role: FOLLOWER
I20260928 02:59:13.449122 21123 consensus_queue.cc:260] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:13.449747 21123 ts_tablet_manager.cc:1428] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:59:13.451930 21145 tablet_replica.cc:330] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: stopping tablet replica
I20260928 02:59:13.452441 21147 tablet_replica.cc:330] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: stopping tablet replica
I20260928 02:59:13.453001 21146 tablet_replica.cc:330] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: stopping tablet replica
I20260928 02:59:13.453595 21147 raft_consensus.cc:2229] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:13.454105 21145 raft_consensus.cc:2229] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:13.454620 21146 raft_consensus.cc:2229] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:59:13.455422 21145 raft_consensus.cc:2258] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:13.456202 21124 tablet_bootstrap.cc:492] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: No bootstrap required, opened a new log
I20260928 02:59:13.456386 21124 ts_tablet_manager.cc:1397] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.007s
I20260928 02:59:13.456408 21147 raft_consensus.cc:2258] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:13.457396 21124 raft_consensus.cc:348] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:13.457540 21146 raft_consensus.cc:2258] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:13.457705 21124 raft_consensus.cc:374] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:13.458024 21124 raft_consensus.cc:729] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cd2557112c43a68222d2fec41cce57, State: Initialized, Role: FOLLOWER
I20260928 02:59:13.458352 21124 consensus_queue.cc:260] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:13.458547 21122 tablet_bootstrap.cc:492] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: No bootstrap required, opened a new log
I20260928 02:59:13.459657 21122 ts_tablet_manager.cc:1397] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260928 02:59:13.459841 21124 ts_tablet_manager.cc:1428] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 02:59:13.460456 21122 raft_consensus.cc:348] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:13.460672 21122 raft_consensus.cc:374] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:13.460858 21122 raft_consensus.cc:729] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c3143914ee47a5af2e53283e380f91, State: Initialized, Role: FOLLOWER
I20260928 02:59:13.461090 21122 consensus_queue.cc:260] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:13.461652 21145 ts_tablet_manager.cc:1905] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:13.462441 21146 ts_tablet_manager.cc:1905] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:13.459363 21147 ts_tablet_manager.cc:1905] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:13.462888 21122 ts_tablet_manager.cc:1428] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 02:59:13.469415 21146 ts_tablet_manager.cc:1918] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:13.469656 21146 log.cc:1196] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/720c35c478274da9989ad1811aa0093e
I20260928 02:59:13.470237 21146 ts_tablet_manager.cc:1939] T 720c35c478274da9989ad1811aa0093e P a5c3143914ee47a5af2e53283e380f91: Deleting consensus metadata
I20260928 02:59:13.470996 21145 ts_tablet_manager.cc:1918] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:13.471153 21145 log.cc:1196] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/720c35c478274da9989ad1811aa0093e
I20260928 02:59:13.471642 21145 ts_tablet_manager.cc:1939] T 720c35c478274da9989ad1811aa0093e P d9cd2557112c43a68222d2fec41cce57: Deleting consensus metadata
I20260928 02:59:13.472079 21147 ts_tablet_manager.cc:1918] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:13.472123 20598 catalog_manager.cc:4841] TS a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541): tablet 720c35c478274da9989ad1811aa0093e (table test-table [id=8e8ab20431a743fe809b1ae5af8104ea]) successfully deleted
I20260928 02:59:13.472231 21147 log.cc:1196] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/720c35c478274da9989ad1811aa0093e
I20260928 02:59:13.472790 21128 raft_consensus.cc:482] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:59:13.472978 21128 raft_consensus.cc:504] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:13.473594 21147 ts_tablet_manager.cc:1939] T 720c35c478274da9989ad1811aa0093e P 01e74da57e85432a890c2241dd3ef877: Deleting consensus metadata
I20260928 02:59:13.474038 20599 catalog_manager.cc:4841] TS d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963): tablet 720c35c478274da9989ad1811aa0093e (table test-table [id=8e8ab20431a743fe809b1ae5af8104ea]) successfully deleted
I20260928 02:59:13.474776 21128 leader_election.cc:288] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191), a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541)
I20260928 02:59:13.475261 20598 catalog_manager.cc:4841] TS 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191): tablet 720c35c478274da9989ad1811aa0093e (table test-table [id=8e8ab20431a743fe809b1ae5af8104ea]) successfully deleted
I20260928 02:59:13.482169 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.475313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:38099 (local address 127.12.182.65:40191)
0928 02:59:13.475487 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.475493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.475881 (+ 388us) server_negotiation.cc:407] Connection header received
0928 02:59:13.476158 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.476165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.476245 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.476304 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:13.476986 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.478125 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.480230 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.481161 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.481316 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.481386 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.481763 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.481894 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:59:13.482213 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.474997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40191 (local address 127.12.182.66:38099)
0928 02:59:13.475797 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.475815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.475979 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.476497 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.476501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.476523 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:13.476825 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.476837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.478255 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.478260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.480086 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.480099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.481369 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.481375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.481567 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.481636 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.481734 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.481864 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":555,"threads_started":1}
I20260928 02:59:13.482918 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a5ff83dc1a42a0a819a07fcb11097f" candidate_uuid: "d9cd2557112c43a68222d2fec41cce57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e74da57e85432a890c2241dd3ef877" is_pre_election: true
I20260928 02:59:13.483094 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.476048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:34423 (local address 127.12.182.67:41541)
0928 02:59:13.476202 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.476207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.476732 (+ 525us) server_negotiation.cc:407] Connection header received
0928 02:59:13.476923 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.476929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.476996 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.477077 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:13.477558 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.478320 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.480763 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.481898 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.482051 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.482115 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.482762 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.482879 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:59:13.483162 20770 raft_consensus.cc:2454] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9cd2557112c43a68222d2fec41cce57 in term 0.
I20260928 02:59:13.483664 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.475932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:41541 (local address 127.12.182.66:34423)
0928 02:59:13.476615 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.476636 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.476766 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.477110 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.477115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.477139 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:13.477418 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.477427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.478464 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.478470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.480585 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.480599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.482391 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.482400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.482586 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.482619 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.483249 (+ 630us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.483363 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":390,"threads_started":1}
I20260928 02:59:13.483686 20840 leader_election.cc:302] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e74da57e85432a890c2241dd3ef877, d9cd2557112c43a68222d2fec41cce57; no voters:
I20260928 02:59:13.484090 21128 raft_consensus.cc:2781] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:59:13.484208 21128 raft_consensus.cc:482] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:59:13.484303 21128 raft_consensus.cc:3037] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:13.484719 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a5ff83dc1a42a0a819a07fcb11097f" candidate_uuid: "d9cd2557112c43a68222d2fec41cce57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c3143914ee47a5af2e53283e380f91" is_pre_election: true
I20260928 02:59:13.485122 21052 raft_consensus.cc:2454] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9cd2557112c43a68222d2fec41cce57 in term 0.
I20260928 02:59:13.487110 21128 raft_consensus.cc:504] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:13.487632 21128 leader_election.cc:288] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [CANDIDATE]: Term 1 election: Requested vote from peers 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191), a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541)
I20260928 02:59:13.488097 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a5ff83dc1a42a0a819a07fcb11097f" candidate_uuid: "d9cd2557112c43a68222d2fec41cce57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c3143914ee47a5af2e53283e380f91"
I20260928 02:59:13.488089 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a5ff83dc1a42a0a819a07fcb11097f" candidate_uuid: "d9cd2557112c43a68222d2fec41cce57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e74da57e85432a890c2241dd3ef877"
I20260928 02:59:13.488271 20770 raft_consensus.cc:3037] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:13.488303 21052 raft_consensus.cc:3037] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:13.490644 21052 raft_consensus.cc:2454] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9cd2557112c43a68222d2fec41cce57 in term 1.
I20260928 02:59:13.490985 20840 leader_election.cc:302] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5c3143914ee47a5af2e53283e380f91, d9cd2557112c43a68222d2fec41cce57; no voters:
I20260928 02:59:13.491079 20770 raft_consensus.cc:2454] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9cd2557112c43a68222d2fec41cce57 in term 1.
I20260928 02:59:13.491250 21128 raft_consensus.cc:2781] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:13.491703 21128 raft_consensus.cc:686] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 1 LEADER]: Becoming Leader. State: Replica: d9cd2557112c43a68222d2fec41cce57, State: Running, Role: LEADER
I20260928 02:59:13.492087 21128 consensus_queue.cc:237] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:13.495332 20613 catalog_manager.cc:5495] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9cd2557112c43a68222d2fec41cce57 (127.12.182.66). New cstate: current_term: 1 leader_uuid: "d9cd2557112c43a68222d2fec41cce57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:59:13 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564351, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.126@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:13.557566 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.544064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.12.182.126:40231)
0928 02:59:13.544294 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.544300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.544850 (+ 550us) server_negotiation.cc:407] Connection header received
0928 02:59:13.545091 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.545095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.545155 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.545233 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:13.546722 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.547423 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.549178 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.549585 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.549717 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.552752 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:13.552756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:13.552760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:13.552786 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:13.555131 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:13.555757 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:13.555762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:13.555764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:13.555872 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:13.556198 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:13.556202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:13.556205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:13.556446 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:13.556576 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.556976 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.557248 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":102}
I20260928 02:59:13.557574 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.544086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40231 (local address 127.0.0.1:51866)
0928 02:59:13.544748 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.544768 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.544977 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.545294 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.545299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.546264 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:13.546593 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.546606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.547545 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.547551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.549059 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.549074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.549754 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.549760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.549904 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.550573 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:13.550590 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:13.552591 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:13.555245 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:13.555256 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:13.555264 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:13.555656 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:13.555966 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:13.555971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:13.555976 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:13.556107 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:13.556561 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:13.556569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:13.556717 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.556850 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.557114 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":359,"threads_started":1}
I20260928 02:59:13.562884 20613 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:51866:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:59:13.563894 20613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:59:13.570148 20889 tablet_service.cc:1467] Processing CreateTablet for tablet b1bc682db05547fdaca15b40b4bd8f7d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5818d47409574021bd4d7065503807ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:59:13.570335 21028 tablet_service.cc:1467] Processing CreateTablet for tablet b1bc682db05547fdaca15b40b4bd8f7d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5818d47409574021bd4d7065503807ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:59:13.570735 20889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bc682db05547fdaca15b40b4bd8f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:13.570575 20750 tablet_service.cc:1467] Processing CreateTablet for tablet b1bc682db05547fdaca15b40b4bd8f7d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5818d47409574021bd4d7065503807ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:59:13.570855 21028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bc682db05547fdaca15b40b4bd8f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:13.571091 20750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bc682db05547fdaca15b40b4bd8f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:13.576107 21122 tablet_bootstrap.cc:492] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: Bootstrap starting.
I20260928 02:59:13.577435 21122 tablet_bootstrap.cc:654] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:13.577716 21123 tablet_bootstrap.cc:492] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: Bootstrap starting.
I20260928 02:59:13.578631 21124 tablet_bootstrap.cc:492] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: Bootstrap starting.
I20260928 02:59:13.579125 21123 tablet_bootstrap.cc:654] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:13.580159 21124 tablet_bootstrap.cc:654] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:13.580717 21123 tablet_bootstrap.cc:492] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: No bootstrap required, opened a new log
I20260928 02:59:13.580853 21123 ts_tablet_manager.cc:1397] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 02:59:13.581365 21123 raft_consensus.cc:348] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.581548 21122 tablet_bootstrap.cc:492] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: No bootstrap required, opened a new log
I20260928 02:59:13.581552 21123 raft_consensus.cc:374] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:13.581642 21123 raft_consensus.cc:729] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e74da57e85432a890c2241dd3ef877, State: Initialized, Role: FOLLOWER
I20260928 02:59:13.581718 21122 ts_tablet_manager.cc:1397] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 02:59:13.581868 21123 consensus_queue.cc:260] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.582378 21123 ts_tablet_manager.cc:1428] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:59:13.582386 21122 raft_consensus.cc:348] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.582607 21122 raft_consensus.cc:374] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:13.582677 21122 raft_consensus.cc:729] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c3143914ee47a5af2e53283e380f91, State: Initialized, Role: FOLLOWER
I20260928 02:59:13.582475 21130 tablet_replica.cc:439] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } }
I20260928 02:59:13.582801 21130 tablet_replica.cc:442] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:59:13.582929 21122 consensus_queue.cc:260] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.583487 21122 ts_tablet_manager.cc:1428] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:59:13.583384 21129 tablet_replica.cc:439] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } }
I20260928 02:59:13.583676 21129 tablet_replica.cc:442] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:59:13.585041 21124 tablet_bootstrap.cc:492] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: No bootstrap required, opened a new log
I20260928 02:59:13.585186 21124 ts_tablet_manager.cc:1397] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20260928 02:59:13.585866 21124 raft_consensus.cc:348] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.586102 21124 raft_consensus.cc:374] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:13.586174 21124 raft_consensus.cc:729] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cd2557112c43a68222d2fec41cce57, State: Initialized, Role: FOLLOWER
I20260928 02:59:13.586395 21124 consensus_queue.cc:260] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.586952 21124 ts_tablet_manager.cc:1428] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:59:13.586820 21128 tablet_replica.cc:439] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } }
I20260928 02:59:13.587098 21128 tablet_replica.cc:442] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 02:59:13.661329 20960 tablet.cc:2369] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:59:13.679127 21099 tablet.cc:2369] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:59:13.695859 20821 tablet.cc:2369] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:59:13.835814 21129 raft_consensus.cc:482] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:59:13.836009 21129 raft_consensus.cc:504] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.836684 21129 leader_election.cc:288] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963), 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191)
I20260928 02:59:13.837212 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bc682db05547fdaca15b40b4bd8f7d" candidate_uuid: "a5c3143914ee47a5af2e53283e380f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e74da57e85432a890c2241dd3ef877" is_pre_election: true
I20260928 02:59:13.837246 20909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bc682db05547fdaca15b40b4bd8f7d" candidate_uuid: "a5c3143914ee47a5af2e53283e380f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9cd2557112c43a68222d2fec41cce57" is_pre_election: true
I20260928 02:59:13.837441 20770 raft_consensus.cc:2454] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5c3143914ee47a5af2e53283e380f91 in term 0.
I20260928 02:59:13.837499 20909 raft_consensus.cc:2454] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5c3143914ee47a5af2e53283e380f91 in term 0.
I20260928 02:59:13.837956 20980 leader_election.cc:302] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01e74da57e85432a890c2241dd3ef877, a5c3143914ee47a5af2e53283e380f91, d9cd2557112c43a68222d2fec41cce57; no voters:
I20260928 02:59:13.838236 21129 raft_consensus.cc:2781] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:59:13.838366 21129 raft_consensus.cc:482] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:59:13.838460 21129 raft_consensus.cc:3037] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:13.840586 21129 raft_consensus.cc:504] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.841073 21129 leader_election.cc:288] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [CANDIDATE]: Term 1 election: Requested vote from peers d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963), 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191)
I20260928 02:59:13.841501 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bc682db05547fdaca15b40b4bd8f7d" candidate_uuid: "a5c3143914ee47a5af2e53283e380f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e74da57e85432a890c2241dd3ef877"
I20260928 02:59:13.841544 20909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bc682db05547fdaca15b40b4bd8f7d" candidate_uuid: "a5c3143914ee47a5af2e53283e380f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9cd2557112c43a68222d2fec41cce57"
I20260928 02:59:13.841749 20909 raft_consensus.cc:3037] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:13.841770 20770 raft_consensus.cc:3037] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:13.844533 20909 raft_consensus.cc:2454] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5c3143914ee47a5af2e53283e380f91 in term 1.
I20260928 02:59:13.844519 20770 raft_consensus.cc:2454] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5c3143914ee47a5af2e53283e380f91 in term 1.
I20260928 02:59:13.844967 20979 leader_election.cc:302] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e74da57e85432a890c2241dd3ef877, a5c3143914ee47a5af2e53283e380f91; no voters:
I20260928 02:59:13.845314 21129 raft_consensus.cc:2781] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:13.845455 21129 raft_consensus.cc:686] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [term 1 LEADER]: Becoming Leader. State: Replica: a5c3143914ee47a5af2e53283e380f91, State: Running, Role: LEADER
I20260928 02:59:13.845759 21129 consensus_queue.cc:237] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } }
I20260928 02:59:13.847587 21142 tablet_replica.cc:439] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: TxnStatusTablet state changed. Reason: New leader a5c3143914ee47a5af2e53283e380f91. Latest consensus state: current_term: 1 leader_uuid: "a5c3143914ee47a5af2e53283e380f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } }
I20260928 02:59:13.847858 21142 tablet_replica.cc:442] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:59:13.848430 21165 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:59:13.848690 21165 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:59:13.848394 20613 catalog_manager.cc:5495] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5c3143914ee47a5af2e53283e380f91 (127.12.182.67). New cstate: current_term: 1 leader_uuid: "a5c3143914ee47a5af2e53283e380f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } health_report { overall_health: UNKNOWN } } }
I20260928 02:59:13.874982 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.869589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.12.182.67:41541)
0928 02:59:13.869782 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.869786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.869804 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:59:13.870064 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.870067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.870125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.870193 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:13.870594 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.871274 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.873125 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.873663 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.873784 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.874222 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:13.874329 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:13.874402 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.874593 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.874668 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:59:13.874971 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.869490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:41541 (local address 127.0.0.1:44216)
0928 02:59:13.869739 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.869760 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.869893 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.870241 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.870247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.870261 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:59:13.870458 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.870470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.871393 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.871399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.872999 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.873014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.873807 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.873813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.873978 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.874045 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:59:13.874462 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:59:13.874480 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.874546 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.874685 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:59:13.878247 20909 raft_consensus.cc:1261] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Refusing update from remote peer a5c3143914ee47a5af2e53283e380f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:13.878305 20770 raft_consensus.cc:1261] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Refusing update from remote peer a5c3143914ee47a5af2e53283e380f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:13.878897 21142 consensus_queue.cc:1035] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:13.879141 21129 consensus_queue.cc:1035] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:13.882447 21130 tablet_replica.cc:439] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: TxnStatusTablet state changed. Reason: New leader a5c3143914ee47a5af2e53283e380f91. Latest consensus state: current_term: 1 leader_uuid: "a5c3143914ee47a5af2e53283e380f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } }
I20260928 02:59:13.882771 21130 tablet_replica.cc:442] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:59:13.883950 21128 tablet_replica.cc:439] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: TxnStatusTablet state changed. Reason: New leader a5c3143914ee47a5af2e53283e380f91. Latest consensus state: current_term: 1 leader_uuid: "a5c3143914ee47a5af2e53283e380f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } }
I20260928 02:59:13.884199 21128 tablet_replica.cc:442] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:59:13.887996 21129 tablet_replica.cc:439] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5c3143914ee47a5af2e53283e380f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } }
I20260928 02:59:13.888499 21128 tablet_replica.cc:439] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5c3143914ee47a5af2e53283e380f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } }
I20260928 02:59:13.888778 21128 tablet_replica.cc:442] T b1bc682db05547fdaca15b40b4bd8f7d P d9cd2557112c43a68222d2fec41cce57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:59:13.890571 21131 tablet_replica.cc:439] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5c3143914ee47a5af2e53283e380f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } }
I20260928 02:59:13.890827 21131 tablet_replica.cc:442] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:59:13.891762 21130 tablet_replica.cc:439] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5c3143914ee47a5af2e53283e380f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } } peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } }
I20260928 02:59:13.891989 21130 tablet_replica.cc:442] T b1bc682db05547fdaca15b40b4bd8f7d P 01e74da57e85432a890c2241dd3ef877: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:59:13.892302 21129 tablet_replica.cc:442] T b1bc682db05547fdaca15b40b4bd8f7d P a5c3143914ee47a5af2e53283e380f91: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:59:13.906634 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.894962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.12.182.66:35963)
0928 02:59:13.901660 (+ 6698us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.901664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.901683 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:59:13.901758 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.901762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.901833 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.901975 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:13.902437 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.903223 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.904507 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.904913 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.905015 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.905431 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:13.906024 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:13.906159 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.906257 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.906332 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6566,"thread_start_us":6444,"threads_started":1}
I20260928 02:59:13.920897 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.915766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:51679 (local address 127.12.182.67:41541)
0928 02:59:13.915934 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.915940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.916462 (+ 522us) server_negotiation.cc:407] Connection header received
0928 02:59:13.916627 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.916632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.916696 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.916764 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:13.917200 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.917992 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.919291 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.919724 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.919830 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.920225 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:13.920339 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:13.920408 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.920532 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.920609 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:59:13.920972 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.915807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:41541 (local address 127.12.182.66:51679)
0928 02:59:13.916359 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.916374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.916500 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.916787 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.916791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.916802 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:59:13.917068 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.917077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.918123 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.918129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.919162 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.919174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.919885 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.919892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.920015 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.920065 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:59:13.920427 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:59:13.920438 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.920505 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.920637 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":303,"threads_started":1}
I20260928 02:59:13.929957 21052 raft_consensus.cc:1261] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Refusing update from remote peer d9cd2557112c43a68222d2fec41cce57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:13.930042 20770 raft_consensus.cc:1261] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Refusing update from remote peer d9cd2557112c43a68222d2fec41cce57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:13.930595 21153 consensus_queue.cc:1035] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:13.931008 21128 consensus_queue.cc:1035] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:13.937089 21137 mvcc.cc:205] Tried to move back new op lower bound from 7334151593689575424 to 7334151591906852864. Current Snapshot: MvccSnapshot[applied={T|T < 7334151593689575424}]
I20260928 02:59:13.965445 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.959590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:53353 (local address 127.12.182.66:35963)
0928 02:59:13.959756 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:13.959762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:13.959918 (+ 156us) server_negotiation.cc:407] Connection header received
0928 02:59:13.960171 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:13.960176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:13.960272 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:13.960329 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:13.961192 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.962165 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.963708 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.964314 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.964417 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.964810 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:13.964903 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:13.964973 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:13.965103 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 02:59:13.965176 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:59:13.965854 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:13.959165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:35963 (local address 127.12.182.67:53353)
0928 02:59:13.959821 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:13.959845 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:13.960020 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:13.960570 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:13.960574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:13.960586 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:59:13.960977 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.960992 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.962308 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.962311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.963587 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:13.963600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:13.964465 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:13.964471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:13.964611 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:13.964663 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:59:13.964995 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:59:13.965006 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:59:13.965076 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:59:13.965198 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":2536,"threads_started":1}
I20260928 02:59:14.004629 20612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:14.004870 20612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:14.008118 20612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7: Sending DeleteTablet for 3 replicas of tablet 55a5ff83dc1a42a0a819a07fcb11097f
I20260928 02:59:14.009055 20750 tablet_service.cc:1514] Processing DeleteTablet for tablet 55a5ff83dc1a42a0a819a07fcb11097f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:14 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:58456
I20260928 02:59:14.009241 20889 tablet_service.cc:1514] Processing DeleteTablet for tablet 55a5ff83dc1a42a0a819a07fcb11097f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:14 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:60630
I20260928 02:59:14.009764 21028 tablet_service.cc:1514] Processing DeleteTablet for tablet 55a5ff83dc1a42a0a819a07fcb11097f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:14 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:44204
I20260928 02:59:14.018009 21185 tablet_replica.cc:330] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: stopping tablet replica
I20260928 02:59:14.018291 21185 raft_consensus.cc:2229] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:14.018567 21185 raft_consensus.cc:2258] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:14.019903 21185 ts_tablet_manager.cc:1905] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:14.024835 21183 tablet_replica.cc:330] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: stopping tablet replica
I20260928 02:59:14.025043 21183 raft_consensus.cc:2229] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:14.025259 21183 raft_consensus.cc:2258] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:14.027127 21183 ts_tablet_manager.cc:1905] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:14.027643 21184 tablet_replica.cc:330] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: stopping tablet replica
I20260928 02:59:14.027899 21184 raft_consensus.cc:2229] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:59:14.028623 21184 raft_consensus.cc:2258] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:14.030328 21184 ts_tablet_manager.cc:1905] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:14.034186 21183 ts_tablet_manager.cc:1918] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:59:14.034299 21183 log.cc:1196] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/55a5ff83dc1a42a0a819a07fcb11097f
I20260928 02:59:14.034709 21183 ts_tablet_manager.cc:1939] T 55a5ff83dc1a42a0a819a07fcb11097f P 01e74da57e85432a890c2241dd3ef877: Deleting consensus metadata
I20260928 02:59:14.035761 20598 catalog_manager.cc:4841] TS 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191): tablet 55a5ff83dc1a42a0a819a07fcb11097f (table test-table [id=ed8e246d865c409aa1c916887224e354]) successfully deleted
I20260928 02:59:14.039090 21185 ts_tablet_manager.cc:1918] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:59:14.039237 21185 log.cc:1196] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/55a5ff83dc1a42a0a819a07fcb11097f
I20260928 02:59:14.039628 21185 ts_tablet_manager.cc:1939] T 55a5ff83dc1a42a0a819a07fcb11097f P a5c3143914ee47a5af2e53283e380f91: Deleting consensus metadata
I20260928 02:59:14.040513 20598 catalog_manager.cc:4841] TS a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541): tablet 55a5ff83dc1a42a0a819a07fcb11097f (table test-table [id=ed8e246d865c409aa1c916887224e354]) successfully deleted
I20260928 02:59:14.043013 21184 ts_tablet_manager.cc:1918] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:59:14.043145 21184 log.cc:1196] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/55a5ff83dc1a42a0a819a07fcb11097f
I20260928 02:59:14.043571 21184 ts_tablet_manager.cc:1939] T 55a5ff83dc1a42a0a819a07fcb11097f P d9cd2557112c43a68222d2fec41cce57: Deleting consensus metadata
I20260928 02:59:14.044673 20599 catalog_manager.cc:4841] TS d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963): tablet 55a5ff83dc1a42a0a819a07fcb11097f (table test-table [id=ed8e246d865c409aa1c916887224e354]) successfully deleted
Sep 28 02:59:14 dist-test-slave-k5rg krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:59:14 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:14 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:59:14.102243 21193 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:14.074567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.12.182.126:40231)
0928 02:59:14.089997 (+ 15430us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.090003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.090025 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:59:14.090091 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.090096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.090180 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.090287 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:14.091992 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.093107 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.095011 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.095608 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.095726 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.101634 (+ 5908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51876: BlockingRecv error: recv got EOF from 127.0.0.1:51876 (error 108)
Metrics: {"server-negotiator.queue_time_us":15279,"thread_start_us":15147,"threads_started":1}
Sep 28 02:59:14 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM
I20260928 02:59:14.125722 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.106579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.12.182.126:40231)
0928 02:59:14.106738 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.106745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.109108 (+ 2363us) server_negotiation.cc:407] Connection header received
0928 02:59:14.109276 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.109282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.109382 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.109485 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:14.111163 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.112258 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.114056 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.114699 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.114817 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.119367 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:14.119374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:14.119379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:14.119421 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:14.122824 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:14.123617 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:14.123622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:14.123626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:14.123770 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:14.124182 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:14.124186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:14.124190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:14.124544 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:14.124784 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:14.125121 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 02:59:14.125345 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:59:14.129474 20612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:59:14.130244 20612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:59:14.137708 20750 tablet_service.cc:1467] Processing CreateTablet for tablet 36fb3e4629f24116b062294e8a59b94a (DEFAULT_TABLE table=test-table [id=99e2af2a125f40fcb087feba90b94f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:14.138211 20750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36fb3e4629f24116b062294e8a59b94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:14.144706 20889 tablet_service.cc:1467] Processing CreateTablet for tablet 36fb3e4629f24116b062294e8a59b94a (DEFAULT_TABLE table=test-table [id=99e2af2a125f40fcb087feba90b94f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:14.145153 20889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36fb3e4629f24116b062294e8a59b94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:14.149164 21028 tablet_service.cc:1467] Processing CreateTablet for tablet 36fb3e4629f24116b062294e8a59b94a (DEFAULT_TABLE table=test-table [id=99e2af2a125f40fcb087feba90b94f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:14.149641 21028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36fb3e4629f24116b062294e8a59b94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:14.152760 21199 tablet_bootstrap.cc:492] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: Bootstrap starting.
I20260928 02:59:14.154420 21199 tablet_bootstrap.cc:654] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:14.157022 21199 tablet_bootstrap.cc:492] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: No bootstrap required, opened a new log
I20260928 02:59:14.157254 21199 ts_tablet_manager.cc:1397] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 02:59:14.158150 21199 raft_consensus.cc:348] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.158444 21202 tablet_bootstrap.cc:492] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91: Bootstrap starting.
I20260928 02:59:14.158528 21199 raft_consensus.cc:374] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:14.158674 21199 raft_consensus.cc:729] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e74da57e85432a890c2241dd3ef877, State: Initialized, Role: FOLLOWER
I20260928 02:59:14.158988 21199 consensus_queue.cc:260] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.159650 21199 ts_tablet_manager.cc:1428] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:59:14.167116 21202 tablet_bootstrap.cc:654] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:14.170441 21202 tablet_bootstrap.cc:492] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91: No bootstrap required, opened a new log
I20260928 02:59:14.170600 21202 ts_tablet_manager.cc:1397] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260928 02:59:14.171429 21202 raft_consensus.cc:348] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.171684 21202 raft_consensus.cc:374] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:14.172716 21200 tablet_bootstrap.cc:492] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57: Bootstrap starting.
I20260928 02:59:14.174934 21202 raft_consensus.cc:729] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c3143914ee47a5af2e53283e380f91, State: Initialized, Role: FOLLOWER
I20260928 02:59:14.175323 21202 consensus_queue.cc:260] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.178182 21202 ts_tablet_manager.cc:1428] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260928 02:59:14.178462 21200 tablet_bootstrap.cc:654] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:14.188949 21200 tablet_bootstrap.cc:492] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57: No bootstrap required, opened a new log
I20260928 02:59:14.189229 21200 ts_tablet_manager.cc:1397] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20260928 02:59:14.190119 21200 raft_consensus.cc:348] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.190433 21200 raft_consensus.cc:374] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:14.190551 21200 raft_consensus.cc:729] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cd2557112c43a68222d2fec41cce57, State: Initialized, Role: FOLLOWER
I20260928 02:59:14.190846 21200 consensus_queue.cc:260] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.191478 21200 ts_tablet_manager.cc:1428] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:59:14.395640 21177 raft_consensus.cc:482] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:59:14.395879 21177 raft_consensus.cc:504] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.396733 21177 leader_election.cc:288] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191), a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541)
I20260928 02:59:14.397332 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36fb3e4629f24116b062294e8a59b94a" candidate_uuid: "d9cd2557112c43a68222d2fec41cce57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c3143914ee47a5af2e53283e380f91" is_pre_election: true
I20260928 02:59:14.397297 20769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36fb3e4629f24116b062294e8a59b94a" candidate_uuid: "d9cd2557112c43a68222d2fec41cce57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e74da57e85432a890c2241dd3ef877" is_pre_election: true
I20260928 02:59:14.397565 21052 raft_consensus.cc:2454] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9cd2557112c43a68222d2fec41cce57 in term 0.
I20260928 02:59:14.397569 20769 raft_consensus.cc:2454] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9cd2557112c43a68222d2fec41cce57 in term 0.
I20260928 02:59:14.398031 20840 leader_election.cc:302] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5c3143914ee47a5af2e53283e380f91, d9cd2557112c43a68222d2fec41cce57; no voters:
I20260928 02:59:14.398329 21177 raft_consensus.cc:2781] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:59:14.398455 21177 raft_consensus.cc:482] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:59:14.398555 21177 raft_consensus.cc:3037] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:14.400780 21177 raft_consensus.cc:504] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.401297 21177 leader_election.cc:288] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [CANDIDATE]: Term 1 election: Requested vote from peers 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191), a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541)
I20260928 02:59:14.401782 20769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36fb3e4629f24116b062294e8a59b94a" candidate_uuid: "d9cd2557112c43a68222d2fec41cce57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e74da57e85432a890c2241dd3ef877"
I20260928 02:59:14.401865 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36fb3e4629f24116b062294e8a59b94a" candidate_uuid: "d9cd2557112c43a68222d2fec41cce57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c3143914ee47a5af2e53283e380f91"
I20260928 02:59:14.401979 20769 raft_consensus.cc:3037] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:14.402064 21052 raft_consensus.cc:3037] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:14.404069 21052 raft_consensus.cc:2454] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9cd2557112c43a68222d2fec41cce57 in term 1.
I20260928 02:59:14.404428 20840 leader_election.cc:302] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5c3143914ee47a5af2e53283e380f91, d9cd2557112c43a68222d2fec41cce57; no voters:
I20260928 02:59:14.404683 21177 raft_consensus.cc:2781] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:14.404814 21177 raft_consensus.cc:686] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 1 LEADER]: Becoming Leader. State: Replica: d9cd2557112c43a68222d2fec41cce57, State: Running, Role: LEADER
I20260928 02:59:14.405036 20769 raft_consensus.cc:2454] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9cd2557112c43a68222d2fec41cce57 in term 1.
I20260928 02:59:14.405092 21177 consensus_queue.cc:237] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.407897 20613 catalog_manager.cc:5495] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9cd2557112c43a68222d2fec41cce57 (127.12.182.66). New cstate: current_term: 1 leader_uuid: "d9cd2557112c43a68222d2fec41cce57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } health_report { overall_health: UNKNOWN } } }
I20260928 02:59:14.446341 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.441536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.12.182.66:35963)
0928 02:59:14.441768 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.441775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.441919 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:59:14.442085 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.442091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.442170 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.442244 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:14.442602 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.443301 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.444579 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.445032 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.445140 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.445554 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:14.445700 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:14.445778 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:14.445920 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:59:14.445998 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:59:14.449064 21052 raft_consensus.cc:1261] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Refusing update from remote peer d9cd2557112c43a68222d2fec41cce57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:14.449100 20769 raft_consensus.cc:1261] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Refusing update from remote peer d9cd2557112c43a68222d2fec41cce57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:14.449688 21177 consensus_queue.cc:1035] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:14.450012 21128 consensus_queue.cc:1035] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:14.460851 20613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:14.461079 20613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:14.463745 20613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7: Sending DeleteTablet for 3 replicas of tablet 36fb3e4629f24116b062294e8a59b94a
I20260928 02:59:14.464401 20750 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fb3e4629f24116b062294e8a59b94a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:14 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:58456
I20260928 02:59:14.464746 20889 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fb3e4629f24116b062294e8a59b94a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:14 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:60630
I20260928 02:59:14.464872 21028 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fb3e4629f24116b062294e8a59b94a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:14 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:44204
I20260928 02:59:14.464987 21183 tablet_replica.cc:330] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: stopping tablet replica
I20260928 02:59:14.465162 21183 raft_consensus.cc:2229] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:14.465301 21183 pending_rounds.cc:62] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: Trying to abort 2 pending ops.
I20260928 02:59:14.465407 21183 pending_rounds.cc:68] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334151595817349120 op_type: WRITE_OP write_request { tablet_id: "36fb3e4629f24116b062294e8a59b94a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\256\262\347\325\006\"8\n\ntest-admin\022*\n 99e2af2a125f40fcb087feba90b94f58\020\001\030\001 \001(\001" signature: "<redacted>""QW\225\344\325Z<\216\331\277)\324\377\201k\232\177l7\350\347\323u\333\313\343\023\255\337\312\276s\024I\007\3239\206]\264\253\035\247G\032?\001\216\261\317\013\210\200\332\247}\364\003rWj|) " signing_key_seq_num: 0 } } request_id { client_id: "fb83921a6cee4bec8a5c5736807aa5a8" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260928 02:59:14.466058 21183 pending_rounds.cc:68] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334151595646021632 op_type: NO_OP noop_request { }
I20260928 02:59:14.466187 21183 raft_consensus.cc:2866] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260928 02:59:14.466293 21183 raft_consensus.cc:2258] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:14.466195 20612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:59:14.466566 21184 tablet_replica.cc:330] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57: stopping tablet replica
W20260928 02:59:14.466703 20612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:59:14.466744 21184 raft_consensus.cc:2229] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:59:14.467000 21184 raft_consensus.cc:2258] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:14.467262 21185 tablet_replica.cc:330] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91: stopping tablet replica
I20260928 02:59:14.467442 21185 raft_consensus.cc:2229] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:14.467648 21185 raft_consensus.cc:2258] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:14.468312 21184 ts_tablet_manager.cc:1905] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:59:14.468822 21208 consensus_queue.cc:1159] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "01e74da57e85432a890c2241dd3ef877" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260928 02:59:14.470425 21185 ts_tablet_manager.cc:1905] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:14.473827 21183 ts_tablet_manager.cc:1905] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:14.473996 21028 tablet_service.cc:1467] Processing CreateTablet for tablet 018f7947b9bd4ed6ad2e87a55191920a (DEFAULT_TABLE table=test-table [id=400e0f7e71f04ec7a7a4083d7d7ffbe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:14.473970 20750 tablet_service.cc:1467] Processing CreateTablet for tablet 018f7947b9bd4ed6ad2e87a55191920a (DEFAULT_TABLE table=test-table [id=400e0f7e71f04ec7a7a4083d7d7ffbe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:14.474375 21028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018f7947b9bd4ed6ad2e87a55191920a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:14.474407 20750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018f7947b9bd4ed6ad2e87a55191920a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:14.477707 21185 ts_tablet_manager.cc:1918] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:14.477852 21185 log.cc:1196] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/36fb3e4629f24116b062294e8a59b94a
I20260928 02:59:14.478274 21185 ts_tablet_manager.cc:1939] T 36fb3e4629f24116b062294e8a59b94a P a5c3143914ee47a5af2e53283e380f91: Deleting consensus metadata
I20260928 02:59:14.479307 20598 catalog_manager.cc:4841] TS a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541): tablet 36fb3e4629f24116b062294e8a59b94a (table test-table [id=99e2af2a125f40fcb087feba90b94f58]) successfully deleted
I20260928 02:59:14.479818 20889 tablet_service.cc:1467] Processing CreateTablet for tablet 018f7947b9bd4ed6ad2e87a55191920a (DEFAULT_TABLE table=test-table [id=400e0f7e71f04ec7a7a4083d7d7ffbe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:14.480139 21183 ts_tablet_manager.cc:1918] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:14.480275 21183 log.cc:1196] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/36fb3e4629f24116b062294e8a59b94a
I20260928 02:59:14.480288 20889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018f7947b9bd4ed6ad2e87a55191920a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:14.480631 21183 ts_tablet_manager.cc:1939] T 36fb3e4629f24116b062294e8a59b94a P 01e74da57e85432a890c2241dd3ef877: Deleting consensus metadata
I20260928 02:59:14.481168 21199 tablet_bootstrap.cc:492] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877: Bootstrap starting.
I20260928 02:59:14.481652 20598 catalog_manager.cc:4841] TS 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191): tablet 36fb3e4629f24116b062294e8a59b94a (table test-table [id=99e2af2a125f40fcb087feba90b94f58]) successfully deleted
I20260928 02:59:14.482410 21184 ts_tablet_manager.cc:1918] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:14.482523 21184 log.cc:1196] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/36fb3e4629f24116b062294e8a59b94a
I20260928 02:59:14.482662 21199 tablet_bootstrap.cc:654] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:14.482781 21184 ts_tablet_manager.cc:1939] T 36fb3e4629f24116b062294e8a59b94a P d9cd2557112c43a68222d2fec41cce57: Deleting consensus metadata
I20260928 02:59:14.483633 20599 catalog_manager.cc:4841] TS d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963): tablet 36fb3e4629f24116b062294e8a59b94a (table test-table [id=99e2af2a125f40fcb087feba90b94f58]) successfully deleted
I20260928 02:59:14.485371 21202 tablet_bootstrap.cc:492] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91: Bootstrap starting.
I20260928 02:59:14.486727 21200 tablet_bootstrap.cc:492] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57: Bootstrap starting.
I20260928 02:59:14.486837 21202 tablet_bootstrap.cc:654] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:14.488338 21200 tablet_bootstrap.cc:654] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:14.497275 21202 tablet_bootstrap.cc:492] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91: No bootstrap required, opened a new log
I20260928 02:59:14.497419 21202 ts_tablet_manager.cc:1397] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260928 02:59:14.497920 21199 tablet_bootstrap.cc:492] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877: No bootstrap required, opened a new log
I20260928 02:59:14.498131 21199 ts_tablet_manager.cc:1397] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.006s
I20260928 02:59:14.498168 21200 tablet_bootstrap.cc:492] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57: No bootstrap required, opened a new log
I20260928 02:59:14.498353 21200 ts_tablet_manager.cc:1397] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260928 02:59:14.498737 21202 raft_consensus.cc:348] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.498813 21199 raft_consensus.cc:348] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.499058 21199 raft_consensus.cc:374] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:14.499143 21199 raft_consensus.cc:729] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e74da57e85432a890c2241dd3ef877, State: Initialized, Role: FOLLOWER
I20260928 02:59:14.499078 21200 raft_consensus.cc:348] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.499392 21202 raft_consensus.cc:374] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:14.499446 21200 raft_consensus.cc:374] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:14.499365 21199 consensus_queue.cc:260] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.499523 21202 raft_consensus.cc:729] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c3143914ee47a5af2e53283e380f91, State: Initialized, Role: FOLLOWER
I20260928 02:59:14.499607 21200 raft_consensus.cc:729] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cd2557112c43a68222d2fec41cce57, State: Initialized, Role: FOLLOWER
I20260928 02:59:14.499961 21199 ts_tablet_manager.cc:1428] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:59:14.499895 21200 consensus_queue.cc:260] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.500541 21200 ts_tablet_manager.cc:1428] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:59:14.501639 21202 consensus_queue.cc:260] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.504038 21202 ts_tablet_manager.cc:1428] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 02:59:14.504436 21130 raft_consensus.cc:482] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:59:14.504606 21130 raft_consensus.cc:504] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.506295 21130 leader_election.cc:288] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963), a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541)
I20260928 02:59:14.512868 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.506856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:36715 (local address 127.12.182.67:41541)
0928 02:59:14.507791 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.507797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.507811 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:59:14.507862 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.507866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.507938 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.508030 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:14.508650 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.509512 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.511286 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.512001 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.512144 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.512214 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:14.512463 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 02:59:14.512591 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":741,"threads_started":1}
I20260928 02:59:14.512869 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.506427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:34359 (local address 127.12.182.66:35963)
0928 02:59:14.506592 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.506598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.507168 (+ 570us) server_negotiation.cc:407] Connection header received
0928 02:59:14.507461 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.507467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.507537 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.507595 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:14.508285 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.509098 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.511262 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.512026 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.512160 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.512228 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:14.512524 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 02:59:14.512643 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:59:14.512989 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.506596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:41541 (local address 127.12.182.65:36715)
0928 02:59:14.507317 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:14.507332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:14.507414 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:14.508060 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:14.508063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:14.508085 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:14.508463 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:14.508475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.509673 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.509678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:14.511156 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:14.511168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.512164 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.512170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:14.512322 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.512351 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:59:14.512428 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:59:14.512559 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":18,"thread_start_us":203,"threads_started":1}
I20260928 02:59:14.513043 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.506429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:35963 (local address 127.12.182.65:34359)
0928 02:59:14.507061 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:14.507085 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:14.507274 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:14.507787 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:14.507791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:14.507816 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:14.508126 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:14.508142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.509366 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.509371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:14.511127 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:14.511139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.512225 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.512231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:14.512384 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.512414 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:59:14.512490 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:59:14.512607 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":53,"thread_start_us":660,"threads_started":1}
I20260928 02:59:14.513628 20909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018f7947b9bd4ed6ad2e87a55191920a" candidate_uuid: "01e74da57e85432a890c2241dd3ef877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9cd2557112c43a68222d2fec41cce57" is_pre_election: true
I20260928 02:59:14.513830 20909 raft_consensus.cc:2454] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e74da57e85432a890c2241dd3ef877 in term 0.
I20260928 02:59:14.513803 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018f7947b9bd4ed6ad2e87a55191920a" candidate_uuid: "01e74da57e85432a890c2241dd3ef877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c3143914ee47a5af2e53283e380f91" is_pre_election: true
I20260928 02:59:14.514009 21052 raft_consensus.cc:2454] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e74da57e85432a890c2241dd3ef877 in term 0.
I20260928 02:59:14.514266 20702 leader_election.cc:302] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e74da57e85432a890c2241dd3ef877, d9cd2557112c43a68222d2fec41cce57; no voters:
I20260928 02:59:14.514652 21130 raft_consensus.cc:2781] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:59:14.514753 21130 raft_consensus.cc:482] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:59:14.514856 21130 raft_consensus.cc:3037] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:14.516902 21130 raft_consensus.cc:504] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.517362 21130 leader_election.cc:288] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [CANDIDATE]: Term 1 election: Requested vote from peers d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963), a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541)
I20260928 02:59:14.517781 20909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018f7947b9bd4ed6ad2e87a55191920a" candidate_uuid: "01e74da57e85432a890c2241dd3ef877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9cd2557112c43a68222d2fec41cce57"
I20260928 02:59:14.517946 20909 raft_consensus.cc:3037] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:14.519733 20909 raft_consensus.cc:2454] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e74da57e85432a890c2241dd3ef877 in term 1.
I20260928 02:59:14.520035 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018f7947b9bd4ed6ad2e87a55191920a" candidate_uuid: "01e74da57e85432a890c2241dd3ef877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c3143914ee47a5af2e53283e380f91"
I20260928 02:59:14.520078 20702 leader_election.cc:302] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e74da57e85432a890c2241dd3ef877, d9cd2557112c43a68222d2fec41cce57; no voters:
I20260928 02:59:14.520183 21052 raft_consensus.cc:3037] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:14.520316 21130 raft_consensus.cc:2781] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:14.520711 21130 raft_consensus.cc:686] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 1 LEADER]: Becoming Leader. State: Replica: 01e74da57e85432a890c2241dd3ef877, State: Running, Role: LEADER
I20260928 02:59:14.521075 21130 consensus_queue.cc:237] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } }
I20260928 02:59:14.522822 21052 raft_consensus.cc:2454] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e74da57e85432a890c2241dd3ef877 in term 1.
I20260928 02:59:14.524513 20612 catalog_manager.cc:5495] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01e74da57e85432a890c2241dd3ef877 (127.12.182.65). New cstate: current_term: 1 leader_uuid: "01e74da57e85432a890c2241dd3ef877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e74da57e85432a890c2241dd3ef877" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 } health_report { overall_health: UNKNOWN } } }
I20260928 02:59:14.546921 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.541339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.12.182.65:40191)
0928 02:59:14.541876 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.541882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.541901 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:59:14.541979 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.541983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.542059 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.542196 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:14.542616 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.543372 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.544650 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.545050 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.545222 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.545683 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:14.546294 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:14.546421 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:14.546547 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:59:14.546624 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":297,"threads_started":1}
I20260928 02:59:14.560266 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.555370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:59943 (local address 127.12.182.67:41541)
0928 02:59:14.555512 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.555517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.556036 (+ 519us) server_negotiation.cc:407] Connection header received
0928 02:59:14.556179 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.556183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.556251 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.556321 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:14.556748 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.557422 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.558741 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.559135 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.559236 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.559618 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:14.559731 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:14.559802 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:14.559921 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 02:59:14.559998 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:59:14.560357 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.555381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:41541 (local address 127.12.182.65:59943)
0928 02:59:14.555931 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:14.555946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:14.556067 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:14.556346 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:14.556350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:14.556360 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:59:14.556617 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:14.556626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.557555 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.557559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:14.558612 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:14.558624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.559290 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.559297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:14.559415 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.559463 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:59:14.559817 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:59:14.559830 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:59:14.559895 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:59:14.560028 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":267,"threads_started":1}
I20260928 02:59:14.568044 21052 raft_consensus.cc:1261] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Refusing update from remote peer 01e74da57e85432a890c2241dd3ef877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:14.568182 20909 raft_consensus.cc:1261] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Refusing update from remote peer 01e74da57e85432a890c2241dd3ef877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:14.568641 21215 consensus_queue.cc:1035] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c3143914ee47a5af2e53283e380f91" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:14.568993 21130 consensus_queue.cc:1035] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9cd2557112c43a68222d2fec41cce57" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:14.592669 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.587595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:33729 (local address 127.12.182.65:40191)
0928 02:59:14.587738 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.587742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.588200 (+ 458us) server_negotiation.cc:407] Connection header received
0928 02:59:14.588364 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.588367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.588424 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.588492 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:14.588889 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.589623 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.591319 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.591803 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.591892 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.592197 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:14.592283 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:14.592342 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:14.592416 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:59:14.592474 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:59:14.592728 21228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.587581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40191 (local address 127.12.182.67:33729)
0928 02:59:14.588123 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:14.588138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:14.588240 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:14.588518 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:14.588522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:14.588534 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:59:14.588763 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:14.588772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.589760 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.589766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:14.591197 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:14.591211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.591927 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.591931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:14.592049 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.592083 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:59:14.592346 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:59:14.592352 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:59:14.592398 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:59:14.592494 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":328,"threads_started":1}
I20260928 02:59:14.622356 20614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:14.622552 20614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:14.625705 20614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ec13e6b4f864225862eb941b63972d7: Sending DeleteTablet for 3 replicas of tablet 018f7947b9bd4ed6ad2e87a55191920a
I20260928 02:59:14.626508 20750 tablet_service.cc:1514] Processing DeleteTablet for tablet 018f7947b9bd4ed6ad2e87a55191920a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:14 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:58456
I20260928 02:59:14.626688 20889 tablet_service.cc:1514] Processing DeleteTablet for tablet 018f7947b9bd4ed6ad2e87a55191920a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:14 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:60630
I20260928 02:59:14.626802 21183 tablet_replica.cc:330] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877: stopping tablet replica
I20260928 02:59:14.626963 21184 tablet_replica.cc:330] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57: stopping tablet replica
I20260928 02:59:14.626972 21183 raft_consensus.cc:2229] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:59:14.627127 21184 raft_consensus.cc:2229] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:14.627306 21184 raft_consensus.cc:2258] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:14.627456 21183 raft_consensus.cc:2258] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:14.627709 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 20685
I20260928 02:59:14.628798 21183 ts_tablet_manager.cc:1905] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:14.630316 21184 ts_tablet_manager.cc:1905] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:14.630540 21028 tablet_service.cc:1514] Processing DeleteTablet for tablet 018f7947b9bd4ed6ad2e87a55191920a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:14 UTC) from {username='oryx', principal='oryx/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:44204
I20260928 02:59:14.631614 21185 tablet_replica.cc:330] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91: stopping tablet replica
I20260928 02:59:14.631779 21185 raft_consensus.cc:2229] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:14.631958 21185 raft_consensus.cc:2258] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:14.633152 21185 ts_tablet_manager.cc:1905] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:14.636925 21184 ts_tablet_manager.cc:1918] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:59:14.644119 21185 ts_tablet_manager.cc:1918] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:59:14.644315 21185 log.cc:1196] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/018f7947b9bd4ed6ad2e87a55191920a
I20260928 02:59:14.644794 21185 ts_tablet_manager.cc:1939] T 018f7947b9bd4ed6ad2e87a55191920a P a5c3143914ee47a5af2e53283e380f91: Deleting consensus metadata
I20260928 02:59:14.645454 21184 log.cc:1196] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/018f7947b9bd4ed6ad2e87a55191920a
I20260928 02:59:14.645963 21184 ts_tablet_manager.cc:1939] T 018f7947b9bd4ed6ad2e87a55191920a P d9cd2557112c43a68222d2fec41cce57: Deleting consensus metadata
I20260928 02:59:14.646425 20598 catalog_manager.cc:4841] TS a5c3143914ee47a5af2e53283e380f91 (127.12.182.67:41541): tablet 018f7947b9bd4ed6ad2e87a55191920a (table test-table [id=400e0f7e71f04ec7a7a4083d7d7ffbe0]) successfully deleted
I20260928 02:59:14.646709 21183 ts_tablet_manager.cc:1918] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:59:14.646858 21183 log.cc:1196] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/018f7947b9bd4ed6ad2e87a55191920a
I20260928 02:59:14.647287 21183 ts_tablet_manager.cc:1939] T 018f7947b9bd4ed6ad2e87a55191920a P 01e74da57e85432a890c2241dd3ef877: Deleting consensus metadata
I20260928 02:59:14.647260 20599 catalog_manager.cc:4841] TS d9cd2557112c43a68222d2fec41cce57 (127.12.182.66:35963): tablet 018f7947b9bd4ed6ad2e87a55191920a (table test-table [id=400e0f7e71f04ec7a7a4083d7d7ffbe0]) successfully deleted
I20260928 02:59:14.648411 20598 catalog_manager.cc:4841] TS 01e74da57e85432a890c2241dd3ef877 (127.12.182.65:40191): tablet 018f7947b9bd4ed6ad2e87a55191920a (table test-table [id=400e0f7e71f04ec7a7a4083d7d7ffbe0]) successfully deleted
Sep 28 02:59:14 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.65@KRBTEST.COM
I20260928 02:59:14.656678 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.628924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.12.182.65:40191)
0928 02:59:14.629092 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.629097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.638896 (+ 9799us) server_negotiation.cc:407] Connection header received
0928 02:59:14.639286 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.639292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.639397 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.639474 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:14.641063 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.642221 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.643984 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.644649 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.644735 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.649793 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:14.649808 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:14.649828 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:14.649876 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:14.653762 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:14.654496 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:14.654505 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:14.654512 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:14.654675 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:14.655043 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:14.655049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:14.655052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:14.655456 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:14.655780 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:14.655991 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:59:14.656220 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:59:14.658318 20815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:14.781536 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 20685
I20260928 02:59:14.801219 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 20824
Sep 28 02:59:14 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.66@KRBTEST.COM
I20260928 02:59:14.814389 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.801911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.12.182.66:35963)
0928 02:59:14.802076 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.802080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.802097 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:59:14.802249 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.802253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.802336 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.802422 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:14.803569 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.804258 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.805377 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.805987 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.806075 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.809748 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:14.809761 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:14.809776 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:14.809807 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:14.812241 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:14.812774 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:14.812780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:14.812782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:14.812906 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:14.813162 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:14.813165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:14.813167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:14.813482 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:14.813830 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:14.813979 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 02:59:14.814142 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:59:14.815003 20954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:14.934592 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 20824
I20260928 02:59:14.952234 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 20963
Sep 28 02:59:14 dist-test-slave-k5rg krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.182.67@KRBTEST.COM
I20260928 02:59:14.966652 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:14.952986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.12.182.67:41541)
0928 02:59:14.953175 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:14.953181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:14.953200 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:59:14.953452 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:14.953456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:14.953570 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:14.953685 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:14.954945 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.955655 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.956859 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:14.957249 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:14.957355 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:14.961490 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:14.961503 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:14.961520 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:14.961552 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:14.964118 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:14.964732 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:14.964741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:14.964745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:14.964878 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:14.965207 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:14.965212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:14.965216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:14.965548 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:14.965851 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:14.966138 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 02:59:14.966340 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:59:14.967236 21093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:15.109031 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 20963
I20260928 02:59:15.128739 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 20582
I20260928 02:59:15.129849 20654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:15.242980 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 20582
2026-09-28T02:59:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5204 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:59:15 dist-test-slave-k5rg krb5kdc[21240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:59:15 dist-test-slave-k5rg krb5kdc[21240](info): set up 2 sockets
Sep 28 02:59:15 dist-test-slave-k5rg krb5kdc[21240](info): commencing operation
krb5kdc: starting...
W20260928 02:59:16.242698 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:59:16 dist-test-slave-k5rg krb5kdc[21240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:59:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:59:16.436421 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33605
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:16.735641 21253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:16.735913 21253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:16.736040 21253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:16.742924 21253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:59:16.743029 21253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:16.743108 21253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:16.743184 21253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:59:16.743248 21253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:59:16.752626 21253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:16.753803 21253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:16.754992 21253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:16.765668 21258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:16.765630 21261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:16.765702 21259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:16.766496 21253 server_base.cc:1034] running on GCE node
I20260928 02:59:16.767159 21253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:16.768358 21253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:16.769556 21253 hybrid_clock.cc:648] HybridClock initialized: now 1790564356769503 us; error 37 us; skew 500 ppm
I20260928 02:59:16.769882 21253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:59:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1600 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:59:16 dist-test-slave-k5rg krb5kdc[21267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:59:16 dist-test-slave-k5rg krb5kdc[21267](info): set up 2 sockets
Sep 28 02:59:16 dist-test-slave-k5rg krb5kdc[21267](info): commencing operation
krb5kdc: starting...
W20260928 02:59:17.834333 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:59:17 dist-test-slave-k5rg krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:59:18.023191 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:45997
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:45997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:18.325224 21280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:18.325474 21280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:18.325614 21280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:18.332669 21280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:59:18.332767 21280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:18.332839 21280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:18.332914 21280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:59:18.332973 21280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:59:18.342607 21280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:45997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:45997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:18.343890 21280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:18.345145 21280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:18.356137 21285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:18.356175 21286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:18.356362 21288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:18.356961 21280 server_base.cc:1034] running on GCE node
I20260928 02:59:18.357685 21280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:18.358950 21280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:18.360146 21280 hybrid_clock.cc:648] HybridClock initialized: now 1790564358360090 us; error 42 us; skew 500 ppm
I20260928 02:59:18.360469 21280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:18 dist-test-slave-k5rg krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:18.365783 21280 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:59:18.367159 21280 webserver.cc:460] Webserver started at http://127.12.182.126:45671/ using document root <none> and password file <none>
I20260928 02:59:18.367781 21280 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:18.367892 21280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:18.368173 21280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:18.370956 21280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "1332f215ccdd4479a4d08874e835b701"
format_stamp: "Formatted at 2026-09-28 02:59:18 on dist-test-slave-k5rg"
server_key: "8b473dd78b84cc49fb2fd77c71053777"
server_key_iv: "0a13aeff12f2e834ed7c538dc939368c"
server_key_version: "encryptionkey@0"
I20260928 02:59:18.371649 21280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "1332f215ccdd4479a4d08874e835b701"
format_stamp: "Formatted at 2026-09-28 02:59:18 on dist-test-slave-k5rg"
server_key: "8b473dd78b84cc49fb2fd77c71053777"
server_key_iv: "0a13aeff12f2e834ed7c538dc939368c"
server_key_version: "encryptionkey@0"
I20260928 02:59:18.376415 21280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:59:18.379565 21295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:18.380781 21280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:59:18.380930 21280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "1332f215ccdd4479a4d08874e835b701"
format_stamp: "Formatted at 2026-09-28 02:59:18 on dist-test-slave-k5rg"
server_key: "8b473dd78b84cc49fb2fd77c71053777"
server_key_iv: "0a13aeff12f2e834ed7c538dc939368c"
server_key_version: "encryptionkey@0"
I20260928 02:59:18.381107 21280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:18.393733 21280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:18.397172 21280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:18.397410 21280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:18.413303 21280 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:45997
I20260928 02:59:18.413311 21346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:45997 every 8 connection(s)
I20260928 02:59:18.414600 21280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:59:18.418162 21347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:18.419832 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 21280
I20260928 02:59:18.420017 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:59:18.420621 13017 external_mini_cluster.cc:1428] Setting key a16d17fda1aee663d105fd565b2f1d5d
I20260928 02:59:18.429154 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701: Bootstrap starting.
I20260928 02:59:18.432636 21347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:18.433837 21347 log.cc:824] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:59:18 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:59:18.436285 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701: No bootstrap required, opened a new log
I20260928 02:59:18.441777 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:18.422426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.12.182.126:45997)
0928 02:59:18.423164 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:18.423175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:18.423220 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:59:18.424835 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:18.424856 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:18.425219 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:18.425543 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:18.427184 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:18.428450 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:18.430133 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:18.430783 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:18.430906 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:18.435138 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:18.435145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:18.435159 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:18.435192 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:18.438007 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:18.438654 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:18.438660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:18.438666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:18.438829 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:18.439185 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:18.439188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:18.439191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:18.439666 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:18.439955 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:18.440364 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 02:59:18.440757 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":346,"threads_started":1}
I20260928 02:59:18.443544 21347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } }
I20260928 02:59:18.443869 21347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:18.443961 21347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1332f215ccdd4479a4d08874e835b701, State: Initialized, Role: FOLLOWER
I20260928 02:59:18.444550 21347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } }
I20260928 02:59:18.444751 21347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:59:18.444870 21347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:59:18.445007 21347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:18.447028 21347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } }
I20260928 02:59:18.447460 21347 leader_election.cc:302] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1332f215ccdd4479a4d08874e835b701; no voters:
I20260928 02:59:18.447957 21347 leader_election.cc:288] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:59:18.448063 21352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:18.448941 21352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 1 LEADER]: Becoming Leader. State: Replica: 1332f215ccdd4479a4d08874e835b701, State: Running, Role: LEADER
I20260928 02:59:18.449455 21352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } }
I20260928 02:59:18.449873 21347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:59:18.452112 21354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1332f215ccdd4479a4d08874e835b701. Latest consensus state: current_term: 1 leader_uuid: "1332f215ccdd4479a4d08874e835b701" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } } }
I20260928 02:59:18.452176 21353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1332f215ccdd4479a4d08874e835b701" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } } }
I20260928 02:59:18.452468 21354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:18.452486 21353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:18.454236 21359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:59:18.459656 21359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:59:18.468032 21359 catalog_manager.cc:1347] Generated new cluster ID: c766582b4ef94f958c727c00378c3dfc
I20260928 02:59:18.468154 21359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:59:18.476961 21359 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:59:18.478698 21359 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:59:18.484858 21359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701: Generated new TSK 0
I20260928 02:59:18.485857 21359 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:59:18.535250 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45997
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:18.839107 21373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:18.839351 21373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:18.839483 21373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:18.846410 21373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:18.846509 21373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:18.846664 21373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:59:18.855887 21373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:18.857052 21373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:18.858429 21373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:18.870473 21378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:18.870518 21381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:18.870438 21379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:18.871272 21373 server_base.cc:1034] running on GCE node
I20260928 02:59:18.871863 21373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:18.872699 21373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:18.873899 21373 hybrid_clock.cc:648] HybridClock initialized: now 1790564358873845 us; error 40 us; skew 500 ppm
I20260928 02:59:18.874178 21373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:18 dist-test-slave-k5rg krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:18.876959 21373 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:59:18.878302 21373 webserver.cc:460] Webserver started at http://127.12.182.65:43275/ using document root <none> and password file <none>
I20260928 02:59:18.878945 21373 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:18.879053 21373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:18.879319 21373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:18.882048 21373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "9bf880a21d9c4c03b80d09a13601db22"
format_stamp: "Formatted at 2026-09-28 02:59:18 on dist-test-slave-k5rg"
server_key: "36af79b6e1b08e3d5dd8b4532ae93a18"
server_key_iv: "2335898bb226cce07de1303bc1c49636"
server_key_version: "encryptionkey@0"
I20260928 02:59:18.882773 21373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "9bf880a21d9c4c03b80d09a13601db22"
format_stamp: "Formatted at 2026-09-28 02:59:18 on dist-test-slave-k5rg"
server_key: "36af79b6e1b08e3d5dd8b4532ae93a18"
server_key_iv: "2335898bb226cce07de1303bc1c49636"
server_key_version: "encryptionkey@0"
I20260928 02:59:18.887779 21373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:59:18.891004 21388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:18.892273 21373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:59:18.892409 21373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "9bf880a21d9c4c03b80d09a13601db22"
format_stamp: "Formatted at 2026-09-28 02:59:18 on dist-test-slave-k5rg"
server_key: "36af79b6e1b08e3d5dd8b4532ae93a18"
server_key_iv: "2335898bb226cce07de1303bc1c49636"
server_key_version: "encryptionkey@0"
I20260928 02:59:18.892571 21373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:18.914570 21373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:18.918056 21373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:18.918295 21373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:18.919126 21373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:18.920303 21373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:18.920375 21373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:18.920480 21373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:18.920529 21373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:18.948621 21373 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:42353
I20260928 02:59:18.948679 21500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:42353 every 8 connection(s)
I20260928 02:59:18.949826 21373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:59:18.953289 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 21373
I20260928 02:59:18.953495 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:59:18.954172 13017 external_mini_cluster.cc:1428] Setting key 1c85539ccb9aa41777f29e7900c31032
Sep 28 02:59:18 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:59:18.968199 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:18.952063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:41969 (local address 127.12.182.126:45997)
0928 02:59:18.952464 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:18.952468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:18.953278 (+ 810us) server_negotiation.cc:407] Connection header received
0928 02:59:18.955190 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:18.955196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:18.955275 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:18.955367 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:18.956852 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:18.957646 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:18.959199 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:18.959607 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:18.959710 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:18.963082 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:18.963087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:18.963090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:18.963120 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:18.965509 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:18.966167 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:18.966172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:18.966174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:18.966287 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:18.966669 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:18.966672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:18.966674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:18.966923 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:18.967049 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:18.967661 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 02:59:18.967866 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":208,"threads_started":1}
I20260928 02:59:18.969045 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:18.952497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:41969)
0928 02:59:18.953139 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:18.953179 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:18.954945 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:18.955590 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:18.955598 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:18.956182 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:18.956697 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:18.956714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:18.957795 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:18.957800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:18.959074 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:18.959086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:18.959807 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:18.959815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:18.960037 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:18.960633 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:18.960654 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:18.962884 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:18.965650 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:18.965658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:18.965672 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:18.966061 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:18.966383 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:18.966388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:18.966392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:18.966561 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:18.967035 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:18.967041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:18.967332 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 02:59:18.967548 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 02:59:18.968022 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":247,"threads_started":1}
I20260928 02:59:18.970383 21501 heartbeater.cc:347] Connected to a master server at 127.12.182.126:45997
I20260928 02:59:18.970649 21501 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:18.971275 21501 heartbeater.cc:511] Master 127.12.182.126:45997 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
I20260928 02:59:18.973158 21312 ts_manager.cc:194] Registered new tserver with Master: 9bf880a21d9c4c03b80d09a13601db22 (127.12.182.65:42353)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
I20260928 02:59:18.974670 21312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:41969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:59:18.996413 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45997
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:19.299417 21507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:19.299681 21507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:19.299770 21507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:19.307766 21507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:19.307864 21507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:19.307971 21507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:59:19.317152 21507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:19.318336 21507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:19.319499 21507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:19.331508 21512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:19.331672 21513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:19.331648 21515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:19.332542 21507 server_base.cc:1034] running on GCE node
I20260928 02:59:19.333097 21507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:19.334030 21507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:19.335215 21507 hybrid_clock.cc:648] HybridClock initialized: now 1790564359335171 us; error 42 us; skew 500 ppm
I20260928 02:59:19.335489 21507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:19 dist-test-slave-k5rg krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:19.338373 21507 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:59:19.339766 21507 webserver.cc:460] Webserver started at http://127.12.182.66:33271/ using document root <none> and password file <none>
I20260928 02:59:19.340407 21507 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:19.340528 21507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:19.340817 21507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:19.343740 21507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "837196f926ed4b949a9de0774a937628"
format_stamp: "Formatted at 2026-09-28 02:59:19 on dist-test-slave-k5rg"
server_key: "f13fc9bd640750ad8fdb546d95c58ecc"
server_key_iv: "ac1a77d1b0f3d54d6972742e15d0745a"
server_key_version: "encryptionkey@0"
I20260928 02:59:19.344489 21507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "837196f926ed4b949a9de0774a937628"
format_stamp: "Formatted at 2026-09-28 02:59:19 on dist-test-slave-k5rg"
server_key: "f13fc9bd640750ad8fdb546d95c58ecc"
server_key_iv: "ac1a77d1b0f3d54d6972742e15d0745a"
server_key_version: "encryptionkey@0"
I20260928 02:59:19.349327 21507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260928 02:59:19.352593 21522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:19.353906 21507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:59:19.354035 21507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "837196f926ed4b949a9de0774a937628"
format_stamp: "Formatted at 2026-09-28 02:59:19 on dist-test-slave-k5rg"
server_key: "f13fc9bd640750ad8fdb546d95c58ecc"
server_key_iv: "ac1a77d1b0f3d54d6972742e15d0745a"
server_key_version: "encryptionkey@0"
I20260928 02:59:19.354184 21507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:19.370103 21507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:19.373596 21507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:19.373853 21507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:19.374670 21507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:19.375856 21507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:19.375924 21507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:19.376014 21507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:19.376055 21507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:19.404661 21507 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:41887
I20260928 02:59:19.404700 21634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:41887 every 8 connection(s)
I20260928 02:59:19.405821 21507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:59:19.414424 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 21507
I20260928 02:59:19.414593 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:59:19.415151 13017 external_mini_cluster.cc:1428] Setting key db15e3974e2d7a87a5f17e47bfefa4e6
Sep 28 02:59:19 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:59:19.425092 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:19.408043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:52679 (local address 127.12.182.126:45997)
0928 02:59:19.408201 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:19.408206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:19.409268 (+ 1062us) server_negotiation.cc:407] Connection header received
0928 02:59:19.411014 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:19.411020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:19.411094 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:19.411178 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:19.412550 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:19.413655 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:19.415700 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:19.416362 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:19.416481 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:19.419759 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:19.419764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:19.419767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:19.419799 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:19.422351 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:19.422992 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:19.422997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:19.423001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:19.423140 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:19.423521 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:19.423526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:19.423529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:19.423775 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:19.423914 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:19.424538 (+ 624us) server_negotiation.cc:299] Negotiation successful
0928 02:59:19.424752 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:59:19.426052 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:19.408478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:52679)
0928 02:59:19.409127 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:19.409162 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:19.410755 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:19.411402 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:19.411418 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:19.411887 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:19.412387 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:19.412403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:19.413802 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:19.413806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:19.415543 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:19.415558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:19.416597 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:19.416605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:19.416883 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:19.417516 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:19.417546 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:19.419562 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:19.422477 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:19.422485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:19.422500 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:19.422883 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:19.423239 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:19.423244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:19.423248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:19.423420 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:19.423902 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:19.423909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:19.424205 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 02:59:19.424437 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 02:59:19.424907 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":242,"threads_started":1}
I20260928 02:59:19.427306 21635 heartbeater.cc:347] Connected to a master server at 127.12.182.126:45997
I20260928 02:59:19.427590 21635 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:19.428252 21635 heartbeater.cc:511] Master 127.12.182.126:45997 requested a full tablet report, sending...
I20260928 02:59:19.429749 21312 ts_manager.cc:194] Registered new tserver with Master: 837196f926ed4b949a9de0774a937628 (127.12.182.66:41887)
I20260928 02:59:19.430536 21312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:52679
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:59:19.460430 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45997
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:19.760645 21640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:19.760897 21640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:19.760993 21640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:19.768038 21640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:19.768136 21640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:19.768244 21640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:59:19.777576 21640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:19.778767 21640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:19.779979 21640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:19.792268 21646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:19.792325 21645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:19.792323 21648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:19.793081 21640 server_base.cc:1034] running on GCE node
I20260928 02:59:19.793654 21640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:19.794513 21640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:19.795709 21640 hybrid_clock.cc:648] HybridClock initialized: now 1790564359795654 us; error 48 us; skew 500 ppm
I20260928 02:59:19.796002 21640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:19 dist-test-slave-k5rg krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:19.798954 21640 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:59:19.800375 21640 webserver.cc:460] Webserver started at http://127.12.182.67:37805/ using document root <none> and password file <none>
I20260928 02:59:19.801002 21640 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:19.801110 21640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:19.801383 21640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:19.804165 21640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "2b210987987b45c4941095c5d4efca61"
format_stamp: "Formatted at 2026-09-28 02:59:19 on dist-test-slave-k5rg"
server_key: "32a82c1631819ec258d0f889ed907719"
server_key_iv: "52cde8a04bd7012843944c9b86277976"
server_key_version: "encryptionkey@0"
I20260928 02:59:19.804920 21640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "2b210987987b45c4941095c5d4efca61"
format_stamp: "Formatted at 2026-09-28 02:59:19 on dist-test-slave-k5rg"
server_key: "32a82c1631819ec258d0f889ed907719"
server_key_iv: "52cde8a04bd7012843944c9b86277976"
server_key_version: "encryptionkey@0"
I20260928 02:59:19.809554 21640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 02:59:19.812676 21655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:19.813910 21640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 02:59:19.814044 21640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "2b210987987b45c4941095c5d4efca61"
format_stamp: "Formatted at 2026-09-28 02:59:19 on dist-test-slave-k5rg"
server_key: "32a82c1631819ec258d0f889ed907719"
server_key_iv: "52cde8a04bd7012843944c9b86277976"
server_key_version: "encryptionkey@0"
I20260928 02:59:19.814206 21640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:19.842427 21640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:19.846241 21640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:19.846503 21640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:19.847450 21640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:19.848697 21640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:19.848771 21640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:19.848894 21640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:19.848955 21640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:19.877583 21640 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:43345
I20260928 02:59:19.877626 21767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:43345 every 8 connection(s)
I20260928 02:59:19.878793 21640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:59:19.887880 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 21640
I20260928 02:59:19.888025 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:59:19.888461 13017 external_mini_cluster.cc:1428] Setting key 1882063c1babb4e872fad2a3c7ba5d33
Sep 28 02:59:19 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:59:19.897533 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:19.881092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:36055 (local address 127.12.182.126:45997)
0928 02:59:19.881258 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:19.881263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:19.882411 (+ 1148us) server_negotiation.cc:407] Connection header received
0928 02:59:19.884146 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:19.884152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:19.884254 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:19.884350 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:19.885847 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:19.886600 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:19.888082 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:19.888518 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:19.888636 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:19.892309 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:19.892314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:19.892317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:19.892348 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:19.894731 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:19.895386 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:19.895391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:19.895395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:19.895536 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:19.895931 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:19.895935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:19.895937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:19.896192 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:19.896326 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:19.896979 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 02:59:19.897194 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:59:19.898427 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:19.881590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:36055)
0928 02:59:19.882267 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:19.882306 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:19.883892 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:19.884578 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:19.884586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:19.885071 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:19.885693 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:19.885712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:19.886742 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:19.886748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:19.887939 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:19.887954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:19.888748 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:19.888756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:19.889042 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:19.889735 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:19.889766 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:19.892120 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:19.894859 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:19.894868 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:19.894882 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:19.895269 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:19.895635 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:19.895640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:19.895644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:19.895820 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:19.896310 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:19.896318 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:19.896644 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 02:59:19.896862 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 02:59:19.897350 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":243,"threads_started":1}
I20260928 02:59:19.899613 21768 heartbeater.cc:347] Connected to a master server at 127.12.182.126:45997
I20260928 02:59:19.899854 21768 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:19.900537 21768 heartbeater.cc:511] Master 127.12.182.126:45997 requested a full tablet report, sending...
I20260928 02:59:19.901851 21312 ts_manager.cc:194] Registered new tserver with Master: 2b210987987b45c4941095c5d4efca61 (127.12.182.67:43345)
I20260928 02:59:19.902760 21312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:36055
I20260928 02:59:19.903461 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab.
Entry for principal oryx/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab.
WARNING: no policy specified for oryx/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.66@KRBTEST.COM" created.
I20260928 02:59:19.977320 21501 heartbeater.cc:503] Master 127.12.182.126:45997 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab.
Entry for principal oryx/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab.
WARNING: no policy specified for oryx/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab.
Entry for principal oryx/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab.
I20260928 02:59:20.032270 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 21373
Sep 28 02:59:20 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:59:20.047554 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:20.033291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.12.182.65:42353)
0928 02:59:20.034031 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:20.034039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:20.034067 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:59:20.034127 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:20.034134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:20.034348 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:20.034483 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:20.035747 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:20.036736 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:20.037985 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:20.038514 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:20.038611 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:20.042512 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:20.042535 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:20.042547 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:20.042580 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:20.045203 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:20.045878 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:20.045885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:20.045887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:20.046003 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:20.046259 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:20.046262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:20.046263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:20.046596 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:20.046812 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:20.047004 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:59:20.047192 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":467,"threads_started":1}
I20260928 02:59:20.048703 21496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:20.128954 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 21373
I20260928 02:59:20.141803 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 21507
Sep 28 02:59:20 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 02:59:20.156087 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:20.142611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.12.182.66:41887)
0928 02:59:20.143109 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:20.143116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:20.143141 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:59:20.143205 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:20.143210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:20.143412 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:20.143543 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:20.144630 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:20.145589 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:20.146801 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:20.147322 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:20.147411 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:20.151087 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:20.151098 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:20.151112 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:20.151144 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:20.153838 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:20.154493 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:20.154498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:20.154501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:20.154624 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:20.154881 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:20.154884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:20.154886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:20.155200 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:20.155414 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:20.155633 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:59:20.155813 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":257,"threads_started":1}
I20260928 02:59:20.157435 21630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:20.237618 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 21507
I20260928 02:59:20.250118 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 21640
Sep 28 02:59:20 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 02:59:20.264745 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:20.251004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.12.182.67:43345)
0928 02:59:20.251539 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:20.251546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:20.251574 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:59:20.251633 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:20.251639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:20.251835 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:20.251955 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:20.253119 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:20.254126 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:20.255302 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:20.255818 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:20.255907 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:20.259877 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:20.259887 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:20.259900 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:20.259931 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:20.262509 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:20.263092 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:20.263097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:20.263100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:20.263205 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:20.263474 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:20.263477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:20.263479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:20.263782 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:20.264020 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:20.264207 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:59:20.264397 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":250,"threads_started":1}
I20260928 02:59:20.265923 21763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:20.346410 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 21640
I20260928 02:59:20.358870 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 21280
I20260928 02:59:20.359961 21342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:20.459981 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 21280
I20260928 02:59:20.476261 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:45997
--webserver_interface=127.12.182.126
--webserver_port=45671
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:45997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:20.777678 21785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:20.777967 21785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:20.778088 21785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:20.785027 21785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:59:20.785133 21785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:20.785208 21785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:20.785288 21785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:59:20.785347 21785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:59:20.800904 21785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:45997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:45997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=45671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:20.802803 21785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:20.804507 21785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:20.816848 21790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:20.816792 21793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:20.816823 21791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:20.818854 21785 server_base.cc:1034] running on GCE node
I20260928 02:59:20.819542 21785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:20.820794 21785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:20.821996 21785 hybrid_clock.cc:648] HybridClock initialized: now 1790564360821936 us; error 40 us; skew 500 ppm
I20260928 02:59:20.822317 21785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:20 dist-test-slave-k5rg krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564360, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:20.825126 21785 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:59:20.826553 21785 webserver.cc:460] Webserver started at http://127.12.182.126:45671/ using document root <none> and password file <none>
I20260928 02:59:20.827203 21785 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:20.827327 21785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:20.832432 21785 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:59:20.835078 21800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:20.836275 21785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:59:20.836429 21785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "1332f215ccdd4479a4d08874e835b701"
format_stamp: "Formatted at 2026-09-28 02:59:18 on dist-test-slave-k5rg"
server_key: "8b473dd78b84cc49fb2fd77c71053777"
server_key_iv: "0a13aeff12f2e834ed7c538dc939368c"
server_key_version: "encryptionkey@0"
I20260928 02:59:20.837335 21785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:20.851186 21785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:20.855116 21785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:20.855343 21785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:20.871451 21785 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:45997
I20260928 02:59:20.871486 21851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:45997 every 8 connection(s)
I20260928 02:59:20.872778 21785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:59:20.879163 21852 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:59:20.882256 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701: Bootstrap starting.
I20260928 02:59:20.883378 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 21785
I20260928 02:59:20.884385 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:42353
--local_ip_for_outbound_sockets=127.12.182.65
--tserver_master_addrs=127.12.182.126:45997
--webserver_port=43275
--webserver_interface=127.12.182.65
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.182.65
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:59:20.902925 21852 log.cc:824] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:20.911558 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:59:20.912031 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701: Bootstrap complete.
I20260928 02:59:20.920562 21852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } }
I20260928 02:59:20.921325 21852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1332f215ccdd4479a4d08874e835b701, State: Initialized, Role: FOLLOWER
I20260928 02:59:20.921931 21852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } }
I20260928 02:59:20.922159 21852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:59:20.922258 21852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:59:20.922413 21852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:59:20.925940 21852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } }
I20260928 02:59:20.926373 21852 leader_election.cc:302] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1332f215ccdd4479a4d08874e835b701; no voters:
I20260928 02:59:20.926944 21852 leader_election.cc:288] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:59:20.927063 21856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:59:20.928194 21856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [term 2 LEADER]: Becoming Leader. State: Replica: 1332f215ccdd4479a4d08874e835b701, State: Running, Role: LEADER
I20260928 02:59:20.928799 21856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } }
I20260928 02:59:20.929052 21852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:59:20.931633 21858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1332f215ccdd4479a4d08874e835b701. Latest consensus state: current_term: 2 leader_uuid: "1332f215ccdd4479a4d08874e835b701" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } } }
I20260928 02:59:20.932034 21858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:20.932576 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1332f215ccdd4479a4d08874e835b701" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1332f215ccdd4479a4d08874e835b701" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45997 } } }
I20260928 02:59:20.932883 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701 [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:20.942780 21864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:59:20.953724 21864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:59:20.955240 21864 catalog_manager.cc:1259] Loaded cluster ID: c766582b4ef94f958c727c00378c3dfc
I20260928 02:59:20.955410 21864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:59:20.959465 21864 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:59:20.961093 21864 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1332f215ccdd4479a4d08874e835b701: Loaded TSK: 0
I20260928 02:59:20.962688 21864 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:59:21.232137 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:21.232411 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:21.232544 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:21.239639 21854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:21.239745 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:21.239918 21854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:59:21.249576 21854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.65.keytab
--principal=oryx/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:42353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=43275
--tserver_master_addrs=127.12.182.126:45997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:21.250808 21854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:21.252024 21854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:21.266016 21878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:21.265995 21879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:21.266561 21881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:21.267009 21854 server_base.cc:1034] running on GCE node
I20260928 02:59:21.267603 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:21.268517 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:21.269695 21854 hybrid_clock.cc:648] HybridClock initialized: now 1790564361269633 us; error 53 us; skew 500 ppm
I20260928 02:59:21.269981 21854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564361, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:21.273017 21854 init.cc:377] Logged in from keytab as oryx/127.12.182.65@KRBTEST.COM (short username oryx)
I20260928 02:59:21.274415 21854 webserver.cc:460] Webserver started at http://127.12.182.65:43275/ using document root <none> and password file <none>
I20260928 02:59:21.275072 21854 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:21.275192 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:21.280465 21854 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:59:21.283134 21888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:21.284464 21854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:59:21.284615 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "9bf880a21d9c4c03b80d09a13601db22"
format_stamp: "Formatted at 2026-09-28 02:59:18 on dist-test-slave-k5rg"
server_key: "36af79b6e1b08e3d5dd8b4532ae93a18"
server_key_iv: "2335898bb226cce07de1303bc1c49636"
server_key_version: "encryptionkey@0"
I20260928 02:59:21.285523 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:21.301448 21854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:21.305377 21854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:21.305642 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:21.306527 21854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:21.307855 21854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:21.307964 21854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:21.308054 21854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:21.308128 21854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:21.338841 21854 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:42353
I20260928 02:59:21.338891 22000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:42353 every 8 connection(s)
I20260928 02:59:21.340267 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:59:21.347009 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 21854
I20260928 02:59:21.347738 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:41887
--local_ip_for_outbound_sockets=127.12.182.66
--tserver_master_addrs=127.12.182.126:45997
--webserver_port=33271
--webserver_interface=127.12.182.66
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.182.66
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:21.357844 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:21.343093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:58547)
0928 02:59:21.343890 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:21.343939 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:21.345574 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:21.346969 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:21.346979 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:21.347688 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:21.348400 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.348425 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.349988 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.349992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.351658 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.351672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.352573 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.352579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.352818 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.353477 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:21.353502 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:21.356918 (+ 3416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":305,"threads_started":1}
W20260928 02:59:21.358451 22001 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:45997 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.182.126:45997: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
W20260928 02:59:21.358654 22003 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:21.343103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:58547 (local address 127.12.182.126:45997)
0928 02:59:21.343744 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:21.343755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:21.344034 (+ 279us) server_negotiation.cc:407] Connection header received
0928 02:59:21.346093 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:21.346106 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:21.346425 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:21.346747 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:21.348602 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.349834 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.351776 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.352356 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.352473 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.357298 (+ 4825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:58547: BlockingRecv error: recv got EOF from 127.12.182.65:58547 (error 108)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":221,"threads_started":1}
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:21.366374 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:21.358985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:34481)
0928 02:59:21.359185 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:21.359199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:21.359280 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:21.359870 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:21.359874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:21.360190 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:21.360317 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.360324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.361378 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.361382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.362329 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.362341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.362992 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.362997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.363161 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.363656 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:21.363673 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:21.366117 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115}
W20260928 02:59:21.366941 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:21.359085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:34481 (local address 127.12.182.126:45997)
0928 02:59:21.359620 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:21.359624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:21.359640 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:59:21.359682 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:21.359685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:21.359736 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:21.359824 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:21.360473 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.361245 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.362462 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.362862 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.362967 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.366381 (+ 3414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:34481: BlockingRecv error: recv got EOF from 127.12.182.65:34481 (error 108)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":303,"threads_started":1}
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:21.373862 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:21.366990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:59121)
0928 02:59:21.367200 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:21.367219 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:21.367306 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:21.367652 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:21.367656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:21.367913 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:21.368049 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.368056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.368935 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.368938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.369864 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.369876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.370517 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.370532 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.370705 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.371187 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:21.371203 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:21.373580 (+ 2377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120}
W20260928 02:59:21.374119 22001 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:59:21.374225 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:21.367268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:59121 (local address 127.12.182.126:45997)
0928 02:59:21.367429 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:21.367433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:21.367445 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:59:21.367482 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:21.367485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:21.367536 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:21.367612 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:21.368170 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.368810 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.369993 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.370391 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.370494 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.373797 (+ 3303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:59121: BlockingRecv error: recv got EOF from 127.12.182.65:59121 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 02:59:21.650753 22005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:21.651013 22005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:21.651151 22005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:21.658025 22005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:21.658130 22005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:21.658272 22005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:59:21.667618 22005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.66.keytab
--principal=oryx/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:41887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=33271
--tserver_master_addrs=127.12.182.126:45997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:21.668782 22005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:21.670017 22005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:21.681934 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:21.681859 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:21.681881 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:21.682909 22005 server_base.cc:1034] running on GCE node
I20260928 02:59:21.683466 22005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:21.684306 22005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:21.685519 22005 hybrid_clock.cc:648] HybridClock initialized: now 1790564361685458 us; error 52 us; skew 500 ppm
I20260928 02:59:21.685829 22005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564361, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:21.688607 22005 init.cc:377] Logged in from keytab as oryx/127.12.182.66@KRBTEST.COM (short username oryx)
I20260928 02:59:21.689961 22005 webserver.cc:460] Webserver started at http://127.12.182.66:33271/ using document root <none> and password file <none>
I20260928 02:59:21.690579 22005 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:21.690685 22005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:21.695609 22005 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:59:21.698155 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:21.699370 22005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:59:21.699502 22005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "837196f926ed4b949a9de0774a937628"
format_stamp: "Formatted at 2026-09-28 02:59:19 on dist-test-slave-k5rg"
server_key: "f13fc9bd640750ad8fdb546d95c58ecc"
server_key_iv: "ac1a77d1b0f3d54d6972742e15d0745a"
server_key_version: "encryptionkey@0"
I20260928 02:59:21.700383 22005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:21.713419 22005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:21.717228 22005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:21.717454 22005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:21.718364 22005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:21.719559 22005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:21.719643 22005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:21.719735 22005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:21.719807 22005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:21.748447 22005 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:41887
I20260928 02:59:21.748510 22133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:41887 every 8 connection(s)
I20260928 02:59:21.749800 22005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:59:21.757622 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 22005
I20260928 02:59:21.758312 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:43345
--local_ip_for_outbound_sockets=127.12.182.67
--tserver_master_addrs=127.12.182.126:45997
--webserver_port=37805
--webserver_interface=127.12.182.67
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.182.67
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:59:21.765581 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:21.752203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:46473 (local address 127.12.182.126:45997)
0928 02:59:21.752384 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:21.752389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:21.753416 (+ 1027us) server_negotiation.cc:407] Connection header received
0928 02:59:21.755182 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:21.755188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:21.755278 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:21.755362 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:21.756753 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.757547 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.759751 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.760159 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.760247 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.765339 (+ 5092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:46473: BlockingRecv error: recv got EOF from 127.12.182.66:46473 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:59:21.766222 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:21.752640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:46473)
0928 02:59:21.753274 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:21.753309 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:21.754938 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:21.755593 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:21.755600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:21.756072 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:21.756600 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.756617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.757736 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.757742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.759615 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.759630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.760353 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.760359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.760603 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.761247 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:21.761273 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:21.765188 (+ 3915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":243,"threads_started":1}
W20260928 02:59:21.766881 22134 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:45997 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.182.126:45997: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:21.774745 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:21.767429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:38357)
0928 02:59:21.767601 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:21.767617 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:21.767701 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:21.767957 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:21.767962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:21.768301 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:21.768464 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.768475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.769358 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.769363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.770649 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.770662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.771253 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.771258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.771419 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.771891 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:21.771916 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:21.774565 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:59:21.775288 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:21.767524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:38357 (local address 127.12.182.126:45997)
0928 02:59:21.767671 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:21.767675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:21.767691 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:59:21.767791 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:21.767794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:21.767850 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:21.767921 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:21.768586 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.769247 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.770759 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.771135 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.771218 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.774789 (+ 3571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:38357: BlockingRecv error: recv got EOF from 127.12.182.66:38357 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:21 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:21.782672 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:21.775314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:36845)
0928 02:59:21.775538 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:21.775556 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:21.775645 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:21.775925 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:21.775928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:21.776172 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:21.776302 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.776308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.777539 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.777542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.778427 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:21.778437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.779337 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.779341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:21.779481 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.780015 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:21.780030 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:21.782407 (+ 2377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260928 02:59:21.782815 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:21.775497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:36845 (local address 127.12.182.126:45997)
0928 02:59:21.775654 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:21.775659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:21.775702 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:59:21.775751 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:21.775756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:21.775823 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:21.775923 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:21.776419 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.777428 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.778551 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:21.779187 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:21.779317 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:21.782610 (+ 3293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:36845: BlockingRecv error: recv got EOF from 127.12.182.66:36845 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 02:59:21.782930 22134 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:59:22.064930 22137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:22.065214 22137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:22.065351 22137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:22.072503 22137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:22.072621 22137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:22.072770 22137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:59:22.082306 22137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/krb5kdc/oryx_127.12.182.67.keytab
--principal=oryx/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:43345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=37805
--tserver_master_addrs=127.12.182.126:45997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:22.083562 22137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:22.084829 22137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:22.097059 22142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:22.097016 22145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:22.097090 22143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:22.097826 22137 server_base.cc:1034] running on GCE node
I20260928 02:59:22.098469 22137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:22.099306 22137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:22.100499 22137 hybrid_clock.cc:648] HybridClock initialized: now 1790564362100457 us; error 28 us; skew 500 ppm
I20260928 02:59:22.100795 22137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564362, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:22.103675 22137 init.cc:377] Logged in from keytab as oryx/127.12.182.67@KRBTEST.COM (short username oryx)
I20260928 02:59:22.105046 22137 webserver.cc:460] Webserver started at http://127.12.182.67:37805/ using document root <none> and password file <none>
I20260928 02:59:22.105700 22137 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:22.105806 22137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:22.110741 22137 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:59:22.113111 22152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:22.114326 22137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:59:22.114446 22137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "2b210987987b45c4941095c5d4efca61"
format_stamp: "Formatted at 2026-09-28 02:59:19 on dist-test-slave-k5rg"
server_key: "32a82c1631819ec258d0f889ed907719"
server_key_iv: "52cde8a04bd7012843944c9b86277976"
server_key_version: "encryptionkey@0"
I20260928 02:59:22.115286 22137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:22.153538 22137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:22.157166 22137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:22.157372 22137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:22.158298 22137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:22.159549 22137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:22.159644 22137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:22.159721 22137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:22.159771 22137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:22.188408 22137 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:43345
I20260928 02:59:22.188503 22264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:43345 every 8 connection(s)
I20260928 02:59:22.189831 22137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:59:22.199996 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 22137
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:59:22.204934 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:22.192155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:41365 (local address 127.12.182.126:45997)
0928 02:59:22.192314 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:22.192318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:22.193391 (+ 1073us) server_negotiation.cc:407] Connection header received
0928 02:59:22.195186 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:22.195191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:22.195270 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:22.195357 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:22.196800 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.197494 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.198968 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.199355 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.199463 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.204708 (+ 5245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:41365: BlockingRecv error: recv got EOF from 127.12.182.67:41365 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:59:22.205672 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:22.192571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:41365)
0928 02:59:22.193233 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:22.193274 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:22.194914 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:22.195600 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:22.195608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:22.196102 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:22.196632 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:22.196648 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.197652 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.197658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:22.198833 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:22.198845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.199558 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.199565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:22.199773 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.200410 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:22.200432 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:22.204539 (+ 4107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":241,"threads_started":1}
W20260928 02:59:22.206284 22265 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:45997 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.182.126:45997: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:22.214720 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:22.206842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:54729)
0928 02:59:22.207066 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:22.207082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:22.207171 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:22.207455 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:22.207460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:22.207793 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:22.207919 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:22.207926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.209192 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.209195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:22.210321 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:22.210335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.210992 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.210998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:22.211179 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.211642 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:22.211665 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:22.214519 (+ 2854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122}
W20260928 02:59:22.214946 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:22.206906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:54729 (local address 127.12.182.126:45997)
0928 02:59:22.207070 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:22.207076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:22.207128 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:59:22.207258 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:22.207263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:22.207329 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:22.207434 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:22.208053 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.209088 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.210455 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.210855 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.210950 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.214701 (+ 3751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:54729: BlockingRecv error: recv got EOF from 127.12.182.67:54729 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:59:22.217307 22268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:22.200958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.12.182.126:45997)
0928 02:59:22.201367 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:22.201373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:22.201745 (+ 372us) server_negotiation.cc:407] Connection header received
0928 02:59:22.201899 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:22.201903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:22.201958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:22.202032 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:22.203739 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.204855 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.206307 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.206809 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.206920 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.209780 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:22.209789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:22.209809 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:22.209856 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:22.213254 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:22.213949 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:22.213958 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:22.213967 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:22.214200 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:22.214593 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:22.214597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:22.214600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:22.215463 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:22.215737 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:22.216587 (+ 850us) server_negotiation.cc:299] Negotiation successful
0928 02:59:22.216938 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":204,"threads_started":1}
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:22.223059 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:22.215317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:54431)
0928 02:59:22.215452 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:22.215469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:22.215554 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:22.215892 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:22.215896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:22.216219 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:22.216388 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:22.216400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.217640 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.217645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:22.218950 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:22.218961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.219794 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.219800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:22.219979 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.220509 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:22.220526 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:22.222875 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:59:22.223202 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:22.215410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:54431 (local address 127.12.182.126:45997)
0928 02:59:22.215684 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:22.215691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:22.215703 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:59:22.215736 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:22.215739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:22.215782 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:22.215854 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:22.216524 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.217502 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.219074 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.219667 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.219770 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.223028 (+ 3258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:54431: BlockingRecv error: recv got EOF from 127.12.182.67:54431 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260928 02:59:22.223287 22265 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:22.382506 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:22.374619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:35461)
0928 02:59:22.375083 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:22.375099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:22.375206 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:22.375470 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:22.375474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:22.375964 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:22.376169 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:22.376182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.377428 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.377433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:22.378395 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:22.378407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.379035 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.379039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:22.379150 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.379639 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:22.379655 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:22.382278 (+ 2623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
W20260928 02:59:22.382678 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:22.374675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:35461 (local address 127.12.182.126:45997)
0928 02:59:22.374827 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:22.374831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:22.375161 (+ 330us) server_negotiation.cc:407] Connection header received
0928 02:59:22.375303 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:22.375306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:22.375366 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:22.375445 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:22.376317 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.377307 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.378500 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.378883 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.378989 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.382448 (+ 3459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:35461: BlockingRecv error: recv got EOF from 127.12.182.65:35461 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:22 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:22.792387 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:22.783475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:36023)
0928 02:59:22.783945 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:22.783961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:22.784071 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:22.784330 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:22.784334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:22.784752 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:22.784975 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:22.784985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.785951 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.785955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:22.786955 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:22.786969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.787965 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.787972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:22.788183 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.788846 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:22.788866 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:22.792066 (+ 3200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
W20260928 02:59:22.792560 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:22.783528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:36023 (local address 127.12.182.126:45997)
0928 02:59:22.783690 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:22.783694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:22.784028 (+ 334us) server_negotiation.cc:407] Connection header received
0928 02:59:22.784161 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:22.784164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:22.784228 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:22.784301 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:22.785084 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.785805 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.787096 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:22.787750 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:22.787877 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:22.792311 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:36023: BlockingRecv error: recv got EOF from 127.12.182.66:36023 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:59:23 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:23 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:23.231549 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:23.223767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:35851)
0928 02:59:23.224216 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:23.224231 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:23.224340 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:23.224608 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:23.224611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:23.225063 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:23.225248 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:23.225256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.226225 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.226229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:23.227160 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:23.227171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.228067 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.228072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:23.228194 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:23.228751 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:23.228767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:23.231328 (+ 2561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
W20260928 02:59:23.231683 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:23.223832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:35851 (local address 127.12.182.126:45997)
0928 02:59:23.224047 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:23.224054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:23.224304 (+ 250us) server_negotiation.cc:407] Connection header received
0928 02:59:23.224431 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:23.224435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:23.224495 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:23.224580 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:23.225357 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.226096 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.227286 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.227879 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.227993 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:23.231498 (+ 3505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:35851: BlockingRecv error: recv got EOF from 127.12.182.67:35851 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:59:23 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:23 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:23.394373 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:23.383386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:55217)
0928 02:59:23.383869 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:23.383916 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:23.384041 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:23.384376 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:23.384380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:23.384766 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:23.384937 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:23.384946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.385963 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.385967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:23.386829 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:23.386840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.387510 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.387516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:23.387669 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:23.391615 (+ 3946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:23.391631 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:23.394139 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":245,"threads_started":1}
W20260928 02:59:23.394526 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:23.383508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:55217 (local address 127.12.182.126:45997)
0928 02:59:23.383723 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:23.383728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:23.384002 (+ 274us) server_negotiation.cc:407] Connection header received
0928 02:59:23.384157 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:23.384163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:23.384223 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:23.384305 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:23.385075 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.385835 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.386970 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.387364 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.387474 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:23.394312 (+ 6838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:55217: BlockingRecv error: recv got EOF from 127.12.182.65:55217 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 02:59:23 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:23 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:23.801798 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:23.793775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:48673)
0928 02:59:23.794261 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:23.794275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:23.794448 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:23.794770 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:23.794775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:23.795188 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:23.795424 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:23.795440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.796519 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.796524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:23.797408 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:23.797419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.798159 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.798164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:23.798314 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:23.798856 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:23.798874 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:23.801452 (+ 2578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":271,"threads_started":1}
W20260928 02:59:23.801937 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:23.793846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:48673 (local address 127.12.182.126:45997)
0928 02:59:23.794023 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:23.794029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:23.794382 (+ 353us) server_negotiation.cc:407] Connection header received
0928 02:59:23.794568 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:23.794574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:23.794639 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:23.794717 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:23.795574 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.796395 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.797539 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:23.798007 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:23.798117 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:23.801688 (+ 3571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:48673: BlockingRecv error: recv got EOF from 127.12.182.66:48673 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 02:59:24 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:24 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:24.240619 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:24.232364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:46081)
0928 02:59:24.232952 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:24.232968 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:24.233086 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:24.233521 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:24.233526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:24.234071 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:24.234291 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:24.234302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.235534 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.235538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:24.236431 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:24.236443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.237101 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.237105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:24.237217 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:24.237799 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:24.237816 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:24.240362 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":370,"threads_started":1}
W20260928 02:59:24.240815 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:24.232511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:46081 (local address 127.12.182.126:45997)
0928 02:59:24.232657 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:24.232661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:24.233042 (+ 381us) server_negotiation.cc:407] Connection header received
0928 02:59:24.233233 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:24.233237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:24.233304 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:24.233395 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:24.234470 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.235400 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.236561 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.236970 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.237058 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:24.240587 (+ 3529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:46081: BlockingRecv error: recv got EOF from 127.12.182.67:46081 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 02:59:24 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:24 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:24.402644 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:24.395194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:54813)
0928 02:59:24.395727 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:24.395742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:24.395851 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:24.396137 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:24.396140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:24.396460 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:24.396639 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:24.396647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.397573 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.397576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:24.398464 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:24.398476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.399168 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.399173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:24.399344 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:24.399863 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:24.399879 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:24.402402 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":280,"threads_started":1}
W20260928 02:59:24.402799 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:24.395290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:54813 (local address 127.12.182.126:45997)
0928 02:59:24.395457 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:24.395461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:24.395804 (+ 343us) server_negotiation.cc:407] Connection header received
0928 02:59:24.395940 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:24.395944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:24.396026 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:24.396111 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:24.396753 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.397469 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.398611 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.399036 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.399141 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:24.402578 (+ 3437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:54813: BlockingRecv error: recv got EOF from 127.12.182.65:54813 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 02:59:24 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:24 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:24.810603 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:24.802700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:48825)
0928 02:59:24.803169 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:24.803185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:24.803300 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:24.803679 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:24.803684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:24.804113 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:24.804334 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:24.804347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.805566 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.805571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:24.806485 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:24.806498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.807062 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.807066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:24.807179 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:24.807739 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:24.807756 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:24.810354 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":228,"threads_started":1}
W20260928 02:59:24.810710 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:24.802814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:48825 (local address 127.12.182.126:45997)
0928 02:59:24.803023 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:24.803029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:24.803259 (+ 230us) server_negotiation.cc:407] Connection header received
0928 02:59:24.803402 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:24.803407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:24.803489 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:24.803593 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:24.804480 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.805436 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.806599 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:24.806963 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:24.807047 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:24.810529 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:48825: BlockingRecv error: recv got EOF from 127.12.182.66:48825 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 02:59:25 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:25 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:25.249516 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:25.241494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:52903)
0928 02:59:25.242010 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:25.242039 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:25.242150 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:25.242460 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:25.242464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:25.242873 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:25.243044 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:25.243053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.244355 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.244359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:25.245454 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:25.245466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.246084 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.246087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:25.246201 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:25.246737 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:25.246753 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:25.249239 (+ 2486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":255,"threads_started":1}
W20260928 02:59:25.249586 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:25.241559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:52903 (local address 127.12.182.126:45997)
0928 02:59:25.241742 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:25.241746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:25.242105 (+ 359us) server_negotiation.cc:407] Connection header received
0928 02:59:25.242239 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:25.242243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:25.242305 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:25.242388 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:25.243170 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.244217 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.245576 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.245980 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.246071 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:25.249401 (+ 3330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:52903: BlockingRecv error: recv got EOF from 127.12.182.67:52903 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 02:59:25 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:25 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:25.411499 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:25.403504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:56015)
0928 02:59:25.403967 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:25.403982 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:25.404123 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:25.404512 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:25.404517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:25.404876 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:25.405029 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:25.405039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.406065 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.406071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:25.407172 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:25.407185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.408010 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.408014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:25.408152 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:25.408668 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:25.408685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:25.411284 (+ 2599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
W20260928 02:59:25.411659 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:25.403554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:56015 (local address 127.12.182.126:45997)
0928 02:59:25.403737 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:25.403742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:25.404056 (+ 314us) server_negotiation.cc:407] Connection header received
0928 02:59:25.404231 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:25.404236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:25.404362 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:25.404447 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:25.405159 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.405938 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.407293 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.407888 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.407986 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:25.411463 (+ 3477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:56015: BlockingRecv error: recv got EOF from 127.12.182.65:56015 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 02:59:25 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:25 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:25.819490 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:25.811381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:40099)
0928 02:59:25.811801 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:25.811817 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:25.811965 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:25.812263 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:25.812267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:25.812602 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:25.812755 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:25.812763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.814103 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.814109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:25.815189 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:25.815202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.815857 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.815862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:25.815981 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:25.816544 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:25.816561 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:25.819157 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":230,"threads_started":1}
W20260928 02:59:25.819612 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:25.811485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:40099 (local address 127.12.182.126:45997)
0928 02:59:25.811736 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:25.811741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:25.811904 (+ 163us) server_negotiation.cc:407] Connection header received
0928 02:59:25.812065 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:25.812068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:25.812130 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:25.812204 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:25.812888 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.813923 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.815323 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:25.815729 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:25.815818 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:25.819374 (+ 3556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:40099: BlockingRecv error: recv got EOF from 127.12.182.66:40099 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 02:59:26 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:26 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:26.258067 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:26.250342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:36585)
0928 02:59:26.250808 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:26.250823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:26.250945 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:26.251285 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:26.251289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:26.251683 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:26.251840 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:26.251850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.252804 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.252808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:26.253808 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:26.253820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.254452 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.254457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:26.254613 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:26.255138 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:26.255154 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:26.257768 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":252,"threads_started":1}
W20260928 02:59:26.258198 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:26.250397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:36585 (local address 127.12.182.126:45997)
0928 02:59:26.250599 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:26.250604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:26.250900 (+ 296us) server_negotiation.cc:407] Connection header received
0928 02:59:26.251075 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:26.251078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:26.251144 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:26.251222 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:26.251977 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.252674 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.253936 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.254324 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.254426 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:26.257985 (+ 3559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:36585: BlockingRecv error: recv got EOF from 127.12.182.67:36585 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:59:26 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:26 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:26.420138 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:26.412458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:58957)
0928 02:59:26.412952 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:26.412966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:26.413086 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:26.413398 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:26.413403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:26.413811 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:26.413970 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:26.413978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.414985 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.414991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:26.415900 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:26.415912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.416539 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.416544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:26.416698 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:26.417232 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:26.417248 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:26.419821 (+ 2573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":237,"threads_started":1}
W20260928 02:59:26.420269 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:26.412522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:58957 (local address 127.12.182.126:45997)
0928 02:59:26.412694 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:26.412701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:26.413046 (+ 345us) server_negotiation.cc:407] Connection header received
0928 02:59:26.413193 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:26.413198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:26.413269 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:26.413346 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:26.414102 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.414851 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.416030 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.416413 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.416515 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:26.420038 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:58957: BlockingRecv error: recv got EOF from 127.12.182.65:58957 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 02:59:26 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:26 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:26.827975 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:26.820382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:34855)
0928 02:59:26.820856 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:26.820872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:26.820979 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:26.821238 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:26.821241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:26.821670 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:26.821824 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:26.821874 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.822927 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.822931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:26.823871 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:26.823883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.824497 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.824501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:26.824618 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:26.825159 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:26.825176 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:26.827705 (+ 2529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
W20260928 02:59:26.828050 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:26.820556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:34855 (local address 127.12.182.126:45997)
0928 02:59:26.820745 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:26.820751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:26.820941 (+ 190us) server_negotiation.cc:407] Connection header received
0928 02:59:26.821072 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:26.821076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:26.821137 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:26.821211 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:26.822002 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.822773 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.823976 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:26.824386 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:26.824469 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:26.827878 (+ 3409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:34855: BlockingRecv error: recv got EOF from 127.12.182.66:34855 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 02:59:27 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:27 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:27.266743 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:27.259078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:48907)
0928 02:59:27.259549 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:27.259565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:27.259683 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:27.259982 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:27.259987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:27.260371 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:27.260533 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:27.260542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.261464 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.261470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:27.262433 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:27.262444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.263101 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.263106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:27.263267 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:27.263785 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:27.263802 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:27.266406 (+ 2604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":235,"threads_started":1}
W20260928 02:59:27.266866 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:27.259173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:48907 (local address 127.12.182.126:45997)
0928 02:59:27.259360 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:27.259367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:27.259642 (+ 275us) server_negotiation.cc:407] Connection header received
0928 02:59:27.259798 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:27.259801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:27.259861 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:27.259936 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:27.260666 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.261338 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.262569 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.262974 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.263088 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:27.266640 (+ 3552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:48907: BlockingRecv error: recv got EOF from 127.12.182.67:48907 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:59:27 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:27 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:27.428560 22285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:27.421058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:48317)
0928 02:59:27.421555 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:27.421571 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:27.421717 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:27.421993 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:27.421996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:27.422347 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:27.422496 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:27.422504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.423555 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.423559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:27.424471 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:27.424483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.425140 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.425145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:27.425267 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:27.425811 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:27.425828 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:27.428343 (+ 2515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":282,"threads_started":1}
W20260928 02:59:27.428758 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:27.421133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:48317 (local address 127.12.182.126:45997)
0928 02:59:27.421275 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:27.421280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:27.421647 (+ 367us) server_negotiation.cc:407] Connection header received
0928 02:59:27.421805 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:27.421808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:27.421881 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:27.421957 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:27.422598 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.423437 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.424579 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.424999 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.425091 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:27.428537 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:48317: BlockingRecv error: recv got EOF from 127.12.182.65:48317 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 02:59:27 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:27 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:27.836282 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:27.828848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:51633)
0928 02:59:27.829256 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:27.829273 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:27.829398 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:27.829713 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:27.829717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:27.830092 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:27.830238 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:27.830247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.831246 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.831250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:27.832139 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:27.832152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.832740 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.832744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:27.832863 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:27.833377 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:27.833393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:27.835940 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":208,"threads_started":1}
W20260928 02:59:27.836347 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:27.828911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:51633 (local address 127.12.182.126:45997)
0928 02:59:27.829095 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:27.829101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:27.829338 (+ 237us) server_negotiation.cc:407] Connection header received
0928 02:59:27.829503 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:27.829507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:27.829576 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:27.829680 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:27.830349 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.831120 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.832248 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:27.832625 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:27.832717 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:27.836121 (+ 3404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:51633: BlockingRecv error: recv got EOF from 127.12.182.66:51633 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 02:59:28 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:28 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:28.275151 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:28.267604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:36297)
0928 02:59:28.268061 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:28.268076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:28.268202 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:28.268513 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:28.268518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:28.268907 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:28.269056 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:28.269065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.270091 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.270095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:28.270947 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:28.270957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.271615 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.271621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:28.271780 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:28.272287 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:28.272303 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:28.274821 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":247,"threads_started":1}
W20260928 02:59:28.275285 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:28.267690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:36297 (local address 127.12.182.126:45997)
0928 02:59:28.267864 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:28.267870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:28.268156 (+ 286us) server_negotiation.cc:407] Connection header received
0928 02:59:28.268315 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:28.268319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:28.268377 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:28.268451 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:28.269189 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.269957 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.271085 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.271477 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.271587 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:28.275064 (+ 3477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:36297: BlockingRecv error: recv got EOF from 127.12.182.67:36297 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 02:59:28 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:28 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:28.440557 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:28.429411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:53801)
0928 02:59:28.430008 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:28.430024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:28.430132 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:28.430446 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:28.430450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:28.430835 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:28.431011 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:28.431020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.431933 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.431937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:28.432874 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:28.432888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.433540 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.433545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:28.433696 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:28.437361 (+ 3665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:28.437380 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:28.440214 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":269,"threads_started":1}
W20260928 02:59:28.440694 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:28.429482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:53801 (local address 127.12.182.126:45997)
0928 02:59:28.429672 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:28.429678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:28.430086 (+ 408us) server_negotiation.cc:407] Connection header received
0928 02:59:28.430222 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:28.430226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:28.430301 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:28.430379 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:28.431117 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.431784 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.432988 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.433419 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.433504 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:28.440453 (+ 6949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:53801: BlockingRecv error: recv got EOF from 127.12.182.65:53801 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 02:59:28 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:28 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:28.845149 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:28.837109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:37397)
0928 02:59:28.837731 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:28.837747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:28.837860 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:28.838139 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:28.838142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:28.838536 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:28.838702 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:28.838711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.839652 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.839657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:28.840616 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:28.840630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.841263 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.841268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:28.841407 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:28.842016 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:28.842034 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:28.844860 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":270,"threads_started":1}
W20260928 02:59:28.845463 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:28.837191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:37397 (local address 127.12.182.126:45997)
0928 02:59:28.837380 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:28.837384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:28.837808 (+ 424us) server_negotiation.cc:407] Connection header received
0928 02:59:28.837952 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:28.837955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:28.838015 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:28.838098 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:28.838806 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.839501 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.840729 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:28.841130 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:28.841234 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:28.845248 (+ 4014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:37397: BlockingRecv error: recv got EOF from 127.12.182.66:37397 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 02:59:29 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:29 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:29.283721 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:29.276036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:54433)
0928 02:59:29.276566 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:29.276582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:29.276708 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:29.277012 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:29.277017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:29.277435 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:29.277620 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:29.277630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.278619 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.278625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:29.279508 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:29.279528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.280164 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.280169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:29.280316 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:29.280840 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:29.280856 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:29.283399 (+ 2543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":231,"threads_started":1}
W20260928 02:59:29.283828 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:29.276098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:54433 (local address 127.12.182.126:45997)
0928 02:59:29.276250 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:29.276257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:29.276666 (+ 409us) server_negotiation.cc:407] Connection header received
0928 02:59:29.276827 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:29.276830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:29.276890 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:29.276965 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:29.277759 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.278488 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.279643 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.280029 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.280131 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:29.283601 (+ 3470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:54433: BlockingRecv error: recv got EOF from 127.12.182.67:54433 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 02:59:29 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:29 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:29.449035 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:29.441416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:45673)
0928 02:59:29.441936 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:29.441952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:29.442058 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:29.442324 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:29.442327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:29.442697 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:29.442843 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:29.442887 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.443816 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.443819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:29.444846 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:29.444865 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.445578 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.445582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:29.445703 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:29.446268 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:29.446286 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:29.448799 (+ 2513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":286,"threads_started":1}
W20260928 02:59:29.449124 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:29.441523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:45673 (local address 127.12.182.126:45997)
0928 02:59:29.441737 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:29.441743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:29.442013 (+ 270us) server_negotiation.cc:407] Connection header received
0928 02:59:29.442143 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:29.442147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:29.442206 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:29.442279 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:29.442982 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.443700 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.444963 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.445442 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.445537 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:29.448968 (+ 3431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:45673: BlockingRecv error: recv got EOF from 127.12.182.65:45673 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:59:29 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:29 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:29.857666 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:29.845997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:51847)
0928 02:59:29.846500 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:29.846516 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:29.846627 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:29.846926 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:29.846929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:29.847299 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:29.847454 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:29.847462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.848658 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.848663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:29.849639 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:29.849653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.850401 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.850407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:29.850536 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:29.854598 (+ 4062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:29.854616 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:29.857376 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
W20260928 02:59:29.857854 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:29.846055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:51847 (local address 127.12.182.126:45997)
0928 02:59:29.846244 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:29.846251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:29.846589 (+ 338us) server_negotiation.cc:407] Connection header received
0928 02:59:29.846721 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:29.846724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:29.846785 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:29.846902 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:29.847580 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.848516 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.849775 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:29.850219 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:29.850315 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:29.857586 (+ 7271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:51847: BlockingRecv error: recv got EOF from 127.12.182.66:51847 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 02:59:30 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:30 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:30.292629 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:30.284690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:42067)
0928 02:59:30.285242 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:30.285257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:30.285382 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:30.285711 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:30.285715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:30.286123 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:30.286267 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:30.286276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.287254 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.287258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:30.288148 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:30.288160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.288880 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.288886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:30.289022 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:30.289642 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:30.289667 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:30.292410 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":296,"threads_started":1}
W20260928 02:59:30.292838 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:30.284789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:42067 (local address 127.12.182.126:45997)
0928 02:59:30.284966 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:30.284973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:30.285343 (+ 370us) server_negotiation.cc:407] Connection header received
0928 02:59:30.285496 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:30.285501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:30.285562 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:30.285687 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:30.286426 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.287118 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.288283 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.288707 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.288811 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:30.292621 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:42067: BlockingRecv error: recv got EOF from 127.12.182.67:42067 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 02:59:30 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:30 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:30.457798 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:30.449945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:48475)
0928 02:59:30.450389 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:30.450405 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:30.450512 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:30.450797 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:30.450800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:30.451169 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:30.451328 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:30.451337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.452280 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.452289 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:30.453387 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:30.453401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.454308 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.454313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:30.454428 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:30.454917 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:30.454933 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:30.457516 (+ 2583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":224,"threads_started":1}
W20260928 02:59:30.457968 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:30.449999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:48475 (local address 127.12.182.126:45997)
0928 02:59:30.450150 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:30.450154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:30.450463 (+ 309us) server_negotiation.cc:407] Connection header received
0928 02:59:30.450608 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:30.450612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:30.450680 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:30.450761 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:30.451431 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.452172 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.453516 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.454184 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.454275 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:30.457745 (+ 3470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:48475: BlockingRecv error: recv got EOF from 127.12.182.65:48475 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 02:59:30 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:30 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:30.866542 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:30.858569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:40907)
0928 02:59:30.859008 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:30.859024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:30.859138 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:30.859482 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:30.859486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:30.859902 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:30.860072 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:30.860081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.860998 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.861002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:30.861948 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:30.861959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.862587 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.862591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:30.862713 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:30.863243 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:30.863260 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:30.866297 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":212,"threads_started":1}
W20260928 02:59:30.866670 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:30.858658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:40907 (local address 127.12.182.126:45997)
0928 02:59:30.858885 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:30.858891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:30.859094 (+ 203us) server_negotiation.cc:407] Connection header received
0928 02:59:30.859226 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:30.859230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:30.859292 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:30.859365 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:30.860182 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.860880 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.862052 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:30.862458 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:30.862556 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:30.866471 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:40907: BlockingRecv error: recv got EOF from 127.12.182.66:40907 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 02:59:31 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:31 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:31.301944 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:31.293580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:37269)
0928 02:59:31.294122 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:31.294138 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:31.294248 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:31.294617 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:31.294621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:31.295011 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:31.295174 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:31.295183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.296198 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.296202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:31.297176 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:31.297189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.297929 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.297934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:31.298070 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:31.298606 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:31.298624 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:31.301704 (+ 3080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":296,"threads_started":1}
W20260928 02:59:31.302090 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:31.293727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:37269 (local address 127.12.182.126:45997)
0928 02:59:31.293918 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:31.293925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:31.294208 (+ 283us) server_negotiation.cc:407] Connection header received
0928 02:59:31.294399 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:31.294403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:31.294479 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:31.294563 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:31.295338 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.296074 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.297288 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.297739 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.297830 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:31.301864 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:37269: BlockingRecv error: recv got EOF from 127.12.182.67:37269 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 02:59:31 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:31 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:31.466652 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:31.458758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:53433)
0928 02:59:31.459239 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:31.459256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:31.459369 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:31.459692 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:31.459695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:31.460075 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:31.460218 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:31.460227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.461284 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.461288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:31.462262 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:31.462274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.462887 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.462891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:31.463023 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:31.463571 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:31.463586 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:31.466351 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":259,"threads_started":1}
W20260928 02:59:31.466781 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:31.458857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:53433 (local address 127.12.182.126:45997)
0928 02:59:31.459051 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:31.459055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:31.459366 (+ 311us) server_negotiation.cc:407] Connection header received
0928 02:59:31.459481 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:31.459485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:31.459555 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:31.459639 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:31.460405 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.461151 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.462368 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.462769 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.462866 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:31.466550 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:53433: BlockingRecv error: recv got EOF from 127.12.182.65:53433 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:59:31 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:31 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:31.875793 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:31.867449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:41665)
0928 02:59:31.867970 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:31.867986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:31.868094 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:31.868358 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:31.868362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:31.868697 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:31.868850 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:31.868858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.869831 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.869836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:31.870766 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:31.870779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.871478 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.871485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:31.871641 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:31.872311 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:31.872336 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:31.875476 (+ 3140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":287,"threads_started":1}
W20260928 02:59:31.875866 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:31.867543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:41665 (local address 127.12.182.126:45997)
0928 02:59:31.867718 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:31.867723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:31.868049 (+ 326us) server_negotiation.cc:407] Connection header received
0928 02:59:31.868177 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:31.868181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:31.868252 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:31.868327 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:31.868963 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.869710 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.870869 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:31.871285 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:31.871367 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:31.875660 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:41665: BlockingRecv error: recv got EOF from 127.12.182.66:41665 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 02:59:32 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:32 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:32.310900 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:32.302925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:58967)
0928 02:59:32.303392 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:32.303411 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:32.303540 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:32.303872 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:32.303877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:32.304319 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:32.304467 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:32.304476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.305690 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.305695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:32.306619 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:32.306630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.307282 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.307288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:32.307438 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:32.307952 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:32.307969 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:32.310667 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":239,"threads_started":1}
W20260928 02:59:32.311081 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:32.303161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:58967 (local address 127.12.182.126:45997)
0928 02:59:32.303378 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:32.303384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:32.303482 (+ 98us) server_negotiation.cc:407] Connection header received
0928 02:59:32.303664 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:32.303670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:32.303734 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:32.303810 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:32.304605 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.305525 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.306759 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.307146 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.307249 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:32.310842 (+ 3593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:58967: BlockingRecv error: recv got EOF from 127.12.182.67:58967 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 02:59:32 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:32 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:32.475513 22300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:32.467614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:40275)
0928 02:59:32.468110 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:32.468126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:32.468239 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:32.468589 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:32.468592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:32.468972 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:32.469169 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:32.469178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.470447 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.470453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:32.471403 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:32.471416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.472054 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.472058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:32.472174 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:32.472736 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:32.472753 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:32.475281 (+ 2528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":256,"threads_started":1}
W20260928 02:59:32.475668 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:32.467704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:40275 (local address 127.12.182.126:45997)
0928 02:59:32.467862 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:32.467866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:32.468195 (+ 329us) server_negotiation.cc:407] Connection header received
0928 02:59:32.468392 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:32.468396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:32.468462 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:32.468546 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:32.469317 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.470313 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.471528 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.471915 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.472022 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:32.475468 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:40275: BlockingRecv error: recv got EOF from 127.12.182.65:40275 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:59:32 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:32 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:32.884794 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:32.876915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:51811)
0928 02:59:32.877587 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:32.877629 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:32.877761 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:32.878058 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:32.878063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:32.878556 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:32.878764 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:32.878775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.879740 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.879743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:32.880754 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:32.880767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.881379 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.881382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:32.881517 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:32.882055 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:32.882071 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:32.884564 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":429,"threads_started":1}
W20260928 02:59:32.884969 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:32.877072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:51811 (local address 127.12.182.126:45997)
0928 02:59:32.877281 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:32.877288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:32.877712 (+ 424us) server_negotiation.cc:407] Connection header received
0928 02:59:32.877869 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:32.877872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:32.877935 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:32.878010 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:32.878919 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.879628 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.880856 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:32.881276 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:32.881365 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:32.884750 (+ 3385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:51811: BlockingRecv error: recv got EOF from 127.12.182.66:51811 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 02:59:33 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:33 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:33.319411 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:33.311749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:36829)
0928 02:59:33.312174 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:33.312190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:33.312302 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:33.312576 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:33.312579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:33.312944 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:33.313110 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:33.313119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.314137 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.314140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:33.315013 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:33.315024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.315648 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.315653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:33.315786 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:33.316312 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:33.316328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:33.319096 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
W20260928 02:59:33.319548 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:33.311821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:36829 (local address 127.12.182.126:45997)
0928 02:59:33.312088 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:33.312094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:33.312294 (+ 200us) server_negotiation.cc:407] Connection header received
0928 02:59:33.312401 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:33.312405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:33.312469 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:33.312549 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:33.313228 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.314041 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.315135 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.315519 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.315607 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:33.319297 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:36829: BlockingRecv error: recv got EOF from 127.12.182.67:36829 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 02:59:33 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:33 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:33.487854 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:33.476398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:51075)
0928 02:59:33.476857 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:33.476874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:33.476987 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:33.477425 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:33.477430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:33.477966 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:33.478164 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:33.478176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.479159 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.479163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:33.480080 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:33.480092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.480750 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.480754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:33.480884 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:33.485000 (+ 4116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:33.485018 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:33.487619 (+ 2601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":242,"threads_started":1}
W20260928 02:59:33.488001 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:33.476445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:51075 (local address 127.12.182.126:45997)
0928 02:59:33.476664 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:33.476671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:33.476949 (+ 278us) server_negotiation.cc:407] Connection header received
0928 02:59:33.477158 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:33.477163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:33.477245 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:33.477369 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:33.478283 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.479028 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.480205 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.480610 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.480717 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:33.487796 (+ 7079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:51075: BlockingRecv error: recv got EOF from 127.12.182.65:51075 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 02:59:33 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:33 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:33.893272 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:33.885594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:44031)
0928 02:59:33.886098 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:33.886115 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:33.886232 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:33.886569 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:33.886572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:33.886934 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:33.887074 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:33.887081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.888074 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.888079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:33.889055 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:33.889068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.889822 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.889826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:33.889949 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:33.890482 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:33.890498 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:33.893019 (+ 2521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":234,"threads_started":1}
W20260928 02:59:33.893397 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:33.885689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:44031 (local address 127.12.182.126:45997)
0928 02:59:33.885889 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:33.885896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:33.886185 (+ 289us) server_negotiation.cc:407] Connection header received
0928 02:59:33.886360 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:33.886364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:33.886447 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:33.886529 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:33.887183 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.887926 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.889177 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:33.889713 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:33.889804 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:33.893213 (+ 3409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:44031: BlockingRecv error: recv got EOF from 127.12.182.66:44031 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 02:59:34 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:34 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:34.328233 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:34.320287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:56041)
0928 02:59:34.320756 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:34.320770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:34.320906 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:34.321215 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:34.321220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:34.321642 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:34.321844 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:34.321857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.323006 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.323010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:34.323963 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:34.323976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.324627 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.324633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:34.324794 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:34.325343 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:34.325360 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:34.327934 (+ 2574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":267,"threads_started":1}
W20260928 02:59:34.328343 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:34.320366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:56041 (local address 127.12.182.126:45997)
0928 02:59:34.320565 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:34.320573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:34.320848 (+ 275us) server_negotiation.cc:407] Connection header received
0928 02:59:34.321023 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:34.321027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:34.321085 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:34.321161 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:34.322014 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.322872 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.324101 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.324501 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.324601 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:34.328137 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:56041: BlockingRecv error: recv got EOF from 127.12.182.67:56041 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 02:59:34 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:34 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:34.496652 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:34.488799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:51813)
0928 02:59:34.489295 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:34.489311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:34.489450 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:34.489799 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:34.489803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:34.490182 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:34.490362 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:34.490384 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.491364 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.491370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:34.492271 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:34.492283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.493041 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.493047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:34.493199 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:34.493783 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:34.493800 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:34.496297 (+ 2497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":249,"threads_started":1}
W20260928 02:59:34.496742 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:34.488879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:51813 (local address 127.12.182.126:45997)
0928 02:59:34.489056 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:34.489062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:34.489391 (+ 329us) server_negotiation.cc:407] Connection header received
0928 02:59:34.489578 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:34.489584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:34.489664 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:34.489742 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:34.490499 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.491227 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.492404 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.492855 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.492984 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:34.496520 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:51813: BlockingRecv error: recv got EOF from 127.12.182.65:51813 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 02:59:34 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:34 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:34.902302 22307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:34.894144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:39445)
0928 02:59:34.894630 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:34.894644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:34.894754 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:34.895008 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:34.895011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:34.895356 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:34.895510 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:34.895518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.896484 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.896488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:34.897393 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:34.897404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.898052 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.898056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:34.898233 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:34.898911 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:34.898945 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:34.901984 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":250,"threads_started":1}
W20260928 02:59:34.902391 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:34.894300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:39445 (local address 127.12.182.126:45997)
0928 02:59:34.894533 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:34.894539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:34.894709 (+ 170us) server_negotiation.cc:407] Connection header received
0928 02:59:34.894841 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:34.894844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:34.894905 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:34.894981 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:34.895634 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.896364 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.897522 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:34.897950 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:34.898042 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:34.902179 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:39445: BlockingRecv error: recv got EOF from 127.12.182.66:39445 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 02:59:35 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:35 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:35.336988 22308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:35.329122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:48715)
0928 02:59:35.329655 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:35.329672 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:35.329798 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:35.330131 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:35.330136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:35.330589 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:35.330744 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:35.330754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.331733 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.331738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:35.332616 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:35.332626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.333319 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.333325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:35.333473 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:35.334033 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:35.334049 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:35.336759 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":249,"threads_started":1}
W20260928 02:59:35.337131 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:35.329204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:48715 (local address 127.12.182.126:45997)
0928 02:59:35.329420 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:35.329427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:35.329744 (+ 317us) server_negotiation.cc:407] Connection header received
0928 02:59:35.329950 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:35.329954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:35.330015 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:35.330091 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:35.330885 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.331608 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.332746 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.333161 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.333261 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:35.336945 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:48715: BlockingRecv error: recv got EOF from 127.12.182.67:48715 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:59:35 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:35 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:35.505712 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:35.497718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:46393)
0928 02:59:35.498202 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:35.498217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:35.498339 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:35.498871 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:35.498874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:35.499293 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:35.499490 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:35.499504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.500606 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.500610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:35.501526 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:35.501539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.502197 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.502201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:35.502314 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:35.502846 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:35.502862 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:35.505426 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":230,"threads_started":1}
W20260928 02:59:35.505846 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:35.497809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:46393 (local address 127.12.182.126:45997)
0928 02:59:35.498041 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:35.498045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:35.498294 (+ 249us) server_negotiation.cc:407] Connection header received
0928 02:59:35.498475 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:35.498479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:35.498547 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:35.498632 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:35.499653 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.500453 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.501659 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.502087 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.502176 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:35.505636 (+ 3460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:46393: BlockingRecv error: recv got EOF from 127.12.182.65:46393 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 28 02:59:35 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:35 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:35.911368 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:35.903208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:41935)
0928 02:59:35.903664 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:35.903678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:35.903789 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:35.904082 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:35.904086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:35.904404 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:35.904575 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:35.904584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.906027 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.906032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:35.906936 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:35.906948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.907585 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.907589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:35.907702 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:35.908252 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:35.908269 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:35.911143 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":200,"threads_started":1}
W20260928 02:59:35.911504 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:35.903270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:41935 (local address 127.12.182.126:45997)
0928 02:59:35.903486 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:35.903492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:35.903747 (+ 255us) server_negotiation.cc:407] Connection header received
0928 02:59:35.903887 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:35.903891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:35.903957 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:35.904031 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:35.904758 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.905890 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.907063 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:35.907468 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:35.907558 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:35.911316 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:41935: BlockingRecv error: recv got EOF from 127.12.182.66:41935 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:59:36 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:36 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.67@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:36.346587 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:36.337840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.67:42963)
0928 02:59:36.338302 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:36.338318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:36.338455 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:36.338763 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:36.338767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:36.339109 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:36.339284 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:36.339294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.340618 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.340622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:36.341765 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:36.341782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.342815 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.342822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:36.343006 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:36.343545 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:36.343562 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:36.346257 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":240,"threads_started":1}
W20260928 02:59:36.346650 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:36.337885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:42963 (local address 127.12.182.126:45997)
0928 02:59:36.338039 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:36.338044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:36.338389 (+ 345us) server_negotiation.cc:407] Connection header received
0928 02:59:36.338547 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:36.338550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:36.338611 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:36.338689 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:36.339442 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.340482 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.341975 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.342642 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.342766 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:36.346453 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.67:42963: BlockingRecv error: recv got EOF from 127.12.182.67:42963 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 02:59:36 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:36 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.65@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:36.514031 22312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:36.506738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.65:55511)
0928 02:59:36.507193 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:36.507209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:36.507321 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:36.507587 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:36.507590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:36.507960 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:36.508121 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:36.508130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.509032 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.509035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:36.509986 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:36.509998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.510611 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.510615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:36.510719 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:36.511205 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:36.511221 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:36.513786 (+ 2565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":253,"threads_started":1}
W20260928 02:59:36.514182 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:36.506799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:55511 (local address 127.12.182.126:45997)
0928 02:59:36.506983 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:36.506989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:36.507276 (+ 287us) server_negotiation.cc:407] Connection header received
0928 02:59:36.507406 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:36.507409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:36.507483 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:36.507557 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:36.508228 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.508931 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.510099 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.510506 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.510593 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:36.513995 (+ 3402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.65:55511: BlockingRecv error: recv got EOF from 127.12.182.65:55511 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 02:59:36 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:59:36 dist-test-slave-k5rg krb5kdc[21267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.182.66@KRBTEST.COM for oryx/127.12.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:59:36.920606 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:36.912406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45997 (local address 127.12.182.66:38983)
0928 02:59:36.913313 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:36.913328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:36.913469 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:36.913824 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:36.913829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:36.914208 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:36.914363 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:36.914373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.915350 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.915355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:36.916248 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:36.916260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.916915 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.916921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:36.917078 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:36.917671 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:36.917694 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:36.920341 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.182.126:45997: Server oryx/127.12.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":669,"threads_started":1}
W20260928 02:59:36.920821 22006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:36.912456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:38983 (local address 127.12.182.126:45997)
0928 02:59:36.912658 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:36.912664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:36.913410 (+ 746us) server_negotiation.cc:407] Connection header received
0928 02:59:36.913577 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:36.913583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:36.913691 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:36.913771 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:36.914505 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.915220 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.916375 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:36.916783 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:36.916886 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:36.920601 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.182.66:38983: BlockingRecv error: recv got EOF from 127.12.182.66:38983 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:59:37.206840 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 21854
I20260928 02:59:37.217947 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:37.207717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.12.182.65:42353)
0928 02:59:37.208212 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:37.208219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:37.208244 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:59:37.208306 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:37.208312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:37.208531 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:37.208687 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:37.210028 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:37.211027 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:37.212233 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:37.212764 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:37.212855 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:37.215112 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:37.215119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:37.215131 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:37.215163 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:37.217421 (+ 2258us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.182.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:59:37.217718 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56892: Request ticket server kudu/127.12.182.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
W20260928 02:59:37.218032 22315 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56892: Request ticket server kudu/127.12.182.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:59:37.218227 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 21854: Not authorized: Client connection negotiation failed: client connection to 127.12.182.65:42353: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.182.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:59:37.218416 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 21854
I20260928 02:59:37.232026 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 22005
I20260928 02:59:37.244026 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:37.233009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.12.182.66:41887)
0928 02:59:37.233542 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:37.233549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:37.233579 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:59:37.233680 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:37.233687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:37.233895 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:37.234049 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:37.235294 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:37.236358 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:37.237574 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:37.238202 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:37.238305 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:37.240565 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:37.240574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:37.240587 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:37.240620 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:37.243403 (+ 2783us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.182.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:59:37.243697 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59848: Request ticket server kudu/127.12.182.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":240,"threads_started":1}
W20260928 02:59:37.244091 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 22005: Not authorized: Client connection negotiation failed: client connection to 127.12.182.66:41887: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.182.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:59:37.244127 22316 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59848: Request ticket server kudu/127.12.182.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:59:37.244218 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 22005
I20260928 02:59:37.257220 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 22137
I20260928 02:59:37.267779 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:37.258085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.12.182.67:43345)
0928 02:59:37.258583 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:37.258590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:37.258614 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:59:37.258668 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:37.258674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:37.258880 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:37.259020 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:37.260191 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:37.261121 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:37.262277 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:37.262786 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:37.262879 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:37.265025 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:37.265032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:37.265044 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:37.265079 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:37.267317 (+ 2238us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.182.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:59:37.267547 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55122: Request ticket server kudu/127.12.182.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":235,"threads_started":1}
W20260928 02:59:37.267875 22317 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55122: Request ticket server kudu/127.12.182.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:59:37.267925 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 22137: Not authorized: Client connection negotiation failed: client connection to 127.12.182.67:43345: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.182.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:59:37.268079 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 22137
I20260928 02:59:37.280575 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 21785
I20260928 02:59:37.281756 21847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:37.394963 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 21785
2026-09-28T02:59:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20553 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:59:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:59:37Z Disabled control of system clock
I20260928 02:59:37.459709 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41811
--webserver_interface=127.12.182.126
--webserver_port=0
--builtin_ntp_servers=127.12.182.84:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:41811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:59:37.759707 22322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:37.759974 22322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:37.760068 22322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:37.767139 22322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:59:37.767225 22322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:37.767285 22322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:37.767361 22322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:59:37.767418 22322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:59:37.776623 22322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:41811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:37.777827 22322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:37.779493 22322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:37.790419 22328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:37.790459 22327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:37.790484 22330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:37.791023 22322 server_base.cc:1034] running on GCE node
I20260928 02:59:37.791683 22322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:37.792963 22322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:37.794144 22322 hybrid_clock.cc:648] HybridClock initialized: now 1790564377794109 us; error 29 us; skew 500 ppm
I20260928 02:59:37.794469 22322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:59:37.796447 22322 webserver.cc:460] Webserver started at http://127.12.182.126:40617/ using document root <none> and password file <none>
I20260928 02:59:37.797055 22322 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:37.797134 22322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:37.797389 22322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:37.800096 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "e03f211807854771aa529de0ef590f6a"
format_stamp: "Formatted at 2026-09-28 02:59:37 on dist-test-slave-k5rg"
server_key: "1924407158e36abc62442369558ea662"
server_key_iv: "471b81e9a8718bdd08c2a96598e5f3db"
server_key_version: "encryptionkey@0"
I20260928 02:59:37.800781 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "e03f211807854771aa529de0ef590f6a"
format_stamp: "Formatted at 2026-09-28 02:59:37 on dist-test-slave-k5rg"
server_key: "1924407158e36abc62442369558ea662"
server_key_iv: "471b81e9a8718bdd08c2a96598e5f3db"
server_key_version: "encryptionkey@0"
I20260928 02:59:37.805472 22322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 02:59:37.808609 22336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:37.809892 22322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:59:37.810029 22322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "e03f211807854771aa529de0ef590f6a"
format_stamp: "Formatted at 2026-09-28 02:59:37 on dist-test-slave-k5rg"
server_key: "1924407158e36abc62442369558ea662"
server_key_iv: "471b81e9a8718bdd08c2a96598e5f3db"
server_key_version: "encryptionkey@0"
I20260928 02:59:37.810191 22322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:37.837620 22322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:37.838420 22322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:37.838654 22322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:37.855051 22322 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:41811
I20260928 02:59:37.855077 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:41811 every 8 connection(s)
I20260928 02:59:37.856312 22322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:59:37.859800 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:37.865670 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 22322
I20260928 02:59:37.865837 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:59:37.866456 13017 external_mini_cluster.cc:1428] Setting key 330e6a5b72c94096486e09437fa48c48
I20260928 02:59:37.869803 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a: Bootstrap starting.
I20260928 02:59:37.872924 22388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:37.874035 22388 log.cc:824] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:37.876813 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a: No bootstrap required, opened a new log
I20260928 02:59:37.879277 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:37.868343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.12.182.126:41811)
0928 02:59:37.869299 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:37.869310 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:37.869350 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:59:37.870962 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:37.870987 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:37.871418 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:37.871716 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:37.872339 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:37.873644 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:37.875197 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:37.875813 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:37.875948 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:37.877089 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:37.877097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:37.877109 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:37.877647 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:37.877720 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:37.877740 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:37.877822 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:37.878050 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:59:37.878338 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":344,"threads_started":1}
I20260928 02:59:37.884152 22388 raft_consensus.cc:348] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f211807854771aa529de0ef590f6a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41811 } }
I20260928 02:59:37.884457 22388 raft_consensus.cc:374] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:37.884545 22388 raft_consensus.cc:729] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e03f211807854771aa529de0ef590f6a, State: Initialized, Role: FOLLOWER
I20260928 02:59:37.885023 22388 consensus_queue.cc:260] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f211807854771aa529de0ef590f6a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41811 } }
I20260928 02:59:37.885219 22388 raft_consensus.cc:388] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:59:37.885332 22388 raft_consensus.cc:482] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:59:37.885476 22388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:37.887501 22388 raft_consensus.cc:504] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f211807854771aa529de0ef590f6a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41811 } }
I20260928 02:59:37.887945 22388 leader_election.cc:302] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e03f211807854771aa529de0ef590f6a; no voters:
I20260928 02:59:37.888466 22388 leader_election.cc:288] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:59:37.888588 22393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:37.889184 22393 raft_consensus.cc:686] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [term 1 LEADER]: Becoming Leader. State: Replica: e03f211807854771aa529de0ef590f6a, State: Running, Role: LEADER
I20260928 02:59:37.889725 22393 consensus_queue.cc:237] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f211807854771aa529de0ef590f6a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41811 } }
I20260928 02:59:37.890304 22388 sys_catalog.cc:564] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:59:37.892266 22395 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e03f211807854771aa529de0ef590f6a. Latest consensus state: current_term: 1 leader_uuid: "e03f211807854771aa529de0ef590f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f211807854771aa529de0ef590f6a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41811 } } }
I20260928 02:59:37.892244 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e03f211807854771aa529de0ef590f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f211807854771aa529de0ef590f6a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41811 } } }
I20260928 02:59:37.892617 22395 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:37.892616 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:37.895190 22400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:59:37.900473 22400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:59:37.908268 22400 catalog_manager.cc:1347] Generated new cluster ID: af4abfedd16c4fd482ecec4413708621
I20260928 02:59:37.908370 22400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:59:37.922219 22400 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:59:37.923372 22400 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:59:37.939818 22400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e03f211807854771aa529de0ef590f6a: Generated new TSK 0
I20260928 02:59:37.940659 22400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:59:37.953392 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41811
--builtin_ntp_servers=127.12.182.84:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:59:38.260524 22412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:38.260762 22412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:38.260840 22412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:38.267719 22412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:38.267808 22412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:38.267912 22412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:59:38.276505 22412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:38.277653 22412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:38.279316 22412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:38.291522 22417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:38.291481 22418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:38.291555 22420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:38.292744 22412 server_base.cc:1034] running on GCE node
I20260928 02:59:38.293277 22412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:38.294176 22412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:38.295347 22412 hybrid_clock.cc:648] HybridClock initialized: now 1790564378295296 us; error 49 us; skew 500 ppm
I20260928 02:59:38.295624 22412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:59:38.297626 22412 webserver.cc:460] Webserver started at http://127.12.182.65:42525/ using document root <none> and password file <none>
I20260928 02:59:38.298249 22412 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:38.298343 22412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:38.298624 22412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:38.301272 22412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "93973642340947a591b1390090d6c62d"
format_stamp: "Formatted at 2026-09-28 02:59:38 on dist-test-slave-k5rg"
server_key: "f741d7b8d7c69cda71b7ceeb88dcc796"
server_key_iv: "b25f73d40b113e252b1cd7fac8739ec1"
server_key_version: "encryptionkey@0"
I20260928 02:59:38.302007 22412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "93973642340947a591b1390090d6c62d"
format_stamp: "Formatted at 2026-09-28 02:59:38 on dist-test-slave-k5rg"
server_key: "f741d7b8d7c69cda71b7ceeb88dcc796"
server_key_iv: "b25f73d40b113e252b1cd7fac8739ec1"
server_key_version: "encryptionkey@0"
I20260928 02:59:38.306574 22412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 02:59:38.309855 22426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:38.311071 22412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:59:38.311241 22412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "93973642340947a591b1390090d6c62d"
format_stamp: "Formatted at 2026-09-28 02:59:38 on dist-test-slave-k5rg"
server_key: "f741d7b8d7c69cda71b7ceeb88dcc796"
server_key_iv: "b25f73d40b113e252b1cd7fac8739ec1"
server_key_version: "encryptionkey@0"
I20260928 02:59:38.311427 22412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:38.336644 22412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:38.337438 22412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:38.337739 22412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:38.338632 22412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:38.340129 22412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:38.340235 22412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:38.340341 22412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:38.340415 22412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:38.369244 22412 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:38557
I20260928 02:59:38.369326 22538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:38557 every 8 connection(s)
I20260928 02:59:38.370820 22412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:59:38.372545 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 22412
I20260928 02:59:38.372720 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:59:38.373425 13017 external_mini_cluster.cc:1428] Setting key dd6bfd92fdecb6f05b9de4c1a2f6edbc
I20260928 02:59:38.376394 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41811
--builtin_ntp_servers=127.12.182.84:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:59:38.384873 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:38.373547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:40465 (local address 127.12.182.126:41811)
0928 02:59:38.373766 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:38.373772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:38.375411 (+ 1639us) server_negotiation.cc:407] Connection header received
0928 02:59:38.377779 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:38.377783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:38.377850 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:38.377929 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:38.379154 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:38.380016 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:38.382049 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:38.382544 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:38.382646 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:38.384173 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:38.384179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:38.384182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:38.384201 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:38.384232 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:38.384243 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:38.384314 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:38.384559 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:59:38.384636 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:59:38.386191 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:38.374145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41811 (local address 127.12.182.65:40465)
0928 02:59:38.375287 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:38.375327 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:38.377473 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:38.378161 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:38.378172 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:38.378292 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:38.379009 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:38.379030 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:38.380167 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:38.380172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:38.381925 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:38.381940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:38.382752 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:38.382758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:38.383033 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:38.383785 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:59:38.383789 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:38.383855 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:59:38.383962 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:38.384337 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:38.384348 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:38.384357 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:59:38.384547 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 02:59:38.384919 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":264,"threads_started":1}
I20260928 02:59:38.387645 22539 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41811
I20260928 02:59:38.387923 22539 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:38.388638 22539 heartbeater.cc:511] Master 127.12.182.126:41811 requested a full tablet report, sending...
I20260928 02:59:38.390826 22353 ts_manager.cc:194] Registered new tserver with Master: 93973642340947a591b1390090d6c62d (127.12.182.65:38557)
I20260928 02:59:38.392831 22353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.65:40465
W20260928 02:59:38.679054 22542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:38.679301 22542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:38.679390 22542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:38.686336 22542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:38.686432 22542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:38.686560 22542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:59:38.695408 22542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:38.696611 22542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:38.698310 22542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:38.710224 22550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:38.710292 22547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:38.710316 22548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:38.711095 22542 server_base.cc:1034] running on GCE node
I20260928 02:59:38.711638 22542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:38.712488 22542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:38.713682 22542 hybrid_clock.cc:648] HybridClock initialized: now 1790564378713616 us; error 51 us; skew 500 ppm
I20260928 02:59:38.713991 22542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:59:38.720238 22542 webserver.cc:460] Webserver started at http://127.12.182.66:37027/ using document root <none> and password file <none>
I20260928 02:59:38.720888 22542 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:38.721027 22542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:38.721320 22542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:38.724048 22542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "1b763d5b51644349a38f4501f625f048"
format_stamp: "Formatted at 2026-09-28 02:59:38 on dist-test-slave-k5rg"
server_key: "8889b6c1dbdce1ab715f47c17c24b4b4"
server_key_iv: "1fa9e0dc8b7501d9e0c3f6d0d967c7da"
server_key_version: "encryptionkey@0"
I20260928 02:59:38.724769 22542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "1b763d5b51644349a38f4501f625f048"
format_stamp: "Formatted at 2026-09-28 02:59:38 on dist-test-slave-k5rg"
server_key: "8889b6c1dbdce1ab715f47c17c24b4b4"
server_key_iv: "1fa9e0dc8b7501d9e0c3f6d0d967c7da"
server_key_version: "encryptionkey@0"
I20260928 02:59:38.729764 22542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:59:38.732949 22556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:38.734345 22542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:59:38.734498 22542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "1b763d5b51644349a38f4501f625f048"
format_stamp: "Formatted at 2026-09-28 02:59:38 on dist-test-slave-k5rg"
server_key: "8889b6c1dbdce1ab715f47c17c24b4b4"
server_key_iv: "1fa9e0dc8b7501d9e0c3f6d0d967c7da"
server_key_version: "encryptionkey@0"
I20260928 02:59:38.734688 22542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:38.752367 22542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:38.753156 22542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:38.753422 22542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:38.754403 22542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:38.755757 22542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:38.755849 22542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:38.755968 22542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:38.756034 22542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:38.784868 22542 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:33829
I20260928 02:59:38.784936 22668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:33829 every 8 connection(s)
I20260928 02:59:38.786142 22542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:59:38.789449 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 22542
I20260928 02:59:38.789620 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:59:38.790145 13017 external_mini_cluster.cc:1428] Setting key a2a39cebf1f6cb815b756deb560e9e9e
I20260928 02:59:38.793778 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41811
--builtin_ntp_servers=127.12.182.84:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:59:38.801093 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:38.788811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:50981 (local address 127.12.182.126:41811)
0928 02:59:38.789014 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:38.789018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:38.791662 (+ 2644us) server_negotiation.cc:407] Connection header received
0928 02:59:38.794040 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:38.794046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:38.794119 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:38.794225 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:38.795715 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:38.796646 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:38.798631 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:38.799034 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:38.799128 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:38.800435 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:38.800440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:38.800443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:38.800459 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:38.800489 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:38.800500 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:38.800565 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:38.800762 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:59:38.800835 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:59:38.802158 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:38.789449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41811 (local address 127.12.182.66:50981)
0928 02:59:38.791492 (+ 2043us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:38.791535 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:38.793650 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:38.794613 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:38.794625 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:38.794738 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:38.795546 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:38.795569 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:38.796840 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:38.796844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:38.798499 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:38.798517 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:38.799227 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:38.799233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:38.799478 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:38.800139 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:59:38.800142 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:38.800188 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:59:38.800272 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:38.800579 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:38.800587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:38.800594 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:59:38.800749 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:59:38.801105 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1523,"thread_start_us":1296,"threads_started":1}
I20260928 02:59:38.803493 22669 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41811
I20260928 02:59:38.803750 22669 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:38.804448 22669 heartbeater.cc:511] Master 127.12.182.126:41811 requested a full tablet report, sending...
I20260928 02:59:38.805775 22353 ts_manager.cc:194] Registered new tserver with Master: 1b763d5b51644349a38f4501f625f048 (127.12.182.66:33829)
I20260928 02:59:38.806766 22353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.66:50981
W20260928 02:59:39.107229 22672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:39.107478 22672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:39.107580 22672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:39.114609 22672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:39.114701 22672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:39.114801 22672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:59:39.123517 22672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:39.124743 22672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:39.126473 22672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:39.138564 22678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:39.138650 22677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:39.138605 22680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:39.139448 22672 server_base.cc:1034] running on GCE node
I20260928 02:59:39.139950 22672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:39.140887 22672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:39.142083 22672 hybrid_clock.cc:648] HybridClock initialized: now 1790564379142032 us; error 47 us; skew 500 ppm
I20260928 02:59:39.142364 22672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:59:39.144407 22672 webserver.cc:460] Webserver started at http://127.12.182.67:46079/ using document root <none> and password file <none>
I20260928 02:59:39.145056 22672 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:39.145153 22672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:39.145437 22672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:39.148066 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "184f3316e25144b596975d080c512961"
format_stamp: "Formatted at 2026-09-28 02:59:39 on dist-test-slave-k5rg"
server_key: "065169d8a56967a352de0ac6afa2eaec"
server_key_iv: "8a37b868daec1c316f9dfc9ddb069cbc"
server_key_version: "encryptionkey@0"
I20260928 02:59:39.148790 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "184f3316e25144b596975d080c512961"
format_stamp: "Formatted at 2026-09-28 02:59:39 on dist-test-slave-k5rg"
server_key: "065169d8a56967a352de0ac6afa2eaec"
server_key_iv: "8a37b868daec1c316f9dfc9ddb069cbc"
server_key_version: "encryptionkey@0"
I20260928 02:59:39.153349 22672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 02:59:39.156431 22686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:39.157650 22672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:59:39.157795 22672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "184f3316e25144b596975d080c512961"
format_stamp: "Formatted at 2026-09-28 02:59:39 on dist-test-slave-k5rg"
server_key: "065169d8a56967a352de0ac6afa2eaec"
server_key_iv: "8a37b868daec1c316f9dfc9ddb069cbc"
server_key_version: "encryptionkey@0"
I20260928 02:59:39.157979 22672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:39.193018 22672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:39.193872 22672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:39.194144 22672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:39.195117 22672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:39.196504 22672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:39.196600 22672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:39.196702 22672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:39.196774 22672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:39.225448 22672 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:36727
I20260928 02:59:39.225500 22798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:36727 every 8 connection(s)
I20260928 02:59:39.226689 22672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:59:39.236182 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 22672
I20260928 02:59:39.236321 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:59:39.236733 13017 external_mini_cluster.cc:1428] Setting key 2c7b43f28f434d8978f420ec8588c0c6
I20260928 02:59:39.238152 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:39.229014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:59429 (local address 127.12.182.126:41811)
0928 02:59:39.229232 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:39.229238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:39.230500 (+ 1262us) server_negotiation.cc:407] Connection header received
0928 02:59:39.232073 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:39.232077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:39.232142 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:39.232227 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:39.233186 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:39.233959 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:39.235459 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:39.235849 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:39.235957 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:39.237280 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:39.237286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:39.237290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:39.237310 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:39.237348 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:39.237364 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:39.237451 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:39.237719 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 02:59:39.237817 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:59:39.239197 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:39.229487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41811 (local address 127.12.182.67:59429)
0928 02:59:39.230347 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:39.230384 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:39.231852 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:39.232413 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:39.232420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:39.232498 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:39.233051 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:39.233068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:39.234134 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:39.234137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:39.235322 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:39.235334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:39.236052 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:39.236059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:39.236269 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:39.236945 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:59:39.236949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:39.236999 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:59:39.237098 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:39.237475 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:39.237485 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:39.237494 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:59:39.237704 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 02:59:39.238033 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":260,"threads_started":1}
I20260928 02:59:39.240361 22799 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41811
I20260928 02:59:39.240617 22799 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:39.241259 22799 heartbeater.cc:511] Master 127.12.182.126:41811 requested a full tablet report, sending...
I20260928 02:59:39.242573 22353 ts_manager.cc:194] Registered new tserver with Master: 184f3316e25144b596975d080c512961 (127.12.182.67:36727)
I20260928 02:59:39.243320 22353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.67:59429
I20260928 02:59:39.251870 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:59:39.257642 22391 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:39.256076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.12.182.126:41811)
0928 02:59:39.256264 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:39.256271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:39.256751 (+ 480us) server_negotiation.cc:407] Connection header received
0928 02:59:39.256807 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:39.256810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:39.256924 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:39.257006 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:39.257357 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44740: BlockingRecv error: recv got EOF from 127.0.0.1:44740 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:59:39.258466 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 22412
I20260928 02:59:39.265636 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:39.259319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.12.182.65:38557)
0928 02:59:39.259798 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:39.259804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:39.259829 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:59:39.259885 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:39.259891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:39.260096 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:39.260211 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:39.260546 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:39.261566 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:39.262828 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:39.263373 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:39.263481 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:39.264310 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:39.264318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:39.264330 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:39.264892 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:39.264937 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:39.264956 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:39.265041 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:39.265153 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 02:59:39.265257 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":238,"threads_started":1}
I20260928 02:59:39.266743 22534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:39.346663 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 22412
I20260928 02:59:39.359522 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 22542
I20260928 02:59:39.367085 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:39.360352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.12.182.66:33829)
0928 02:59:39.360840 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:39.360853 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:39.360879 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:59:39.360936 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:39.360942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:39.361168 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:39.361304 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:39.361719 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:39.362704 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:39.363878 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:39.364407 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:39.364526 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:39.365572 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:39.365583 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:39.365627 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:39.366338 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:39.366385 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:39.366403 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:39.366486 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:39.366617 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:59:39.366741 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":221,"threads_started":1}
I20260928 02:59:39.368222 22664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:39.447602 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 22542
I20260928 02:59:39.460517 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 22672
I20260928 02:59:39.468133 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:39.461426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.12.182.67:36727)
0928 02:59:39.462057 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:39.462064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:39.462092 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:59:39.462150 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:39.462156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:39.462362 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:39.462488 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:39.462882 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:39.463980 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:39.465180 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:39.465912 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:39.466008 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:39.466897 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:39.466905 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:39.466917 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:39.467473 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:39.467517 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:39.467538 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:39.467618 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:39.467745 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:59:39.467844 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":335,"threads_started":1}
I20260928 02:59:39.469220 22794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:39.547657 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 22672
I20260928 02:59:39.560562 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 22322
I20260928 02:59:39.561817 22383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:39.659646 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 22322
2026-09-28T02:59:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2260 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:59:39Z Disabled control of system clock
I20260928 02:59:39.720389 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41047
--webserver_interface=127.12.182.126
--webserver_port=0
--builtin_ntp_servers=127.12.182.84:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:41047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:59:40.021348 22818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:40.021628 22818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:40.021735 22818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:40.028861 22818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:59:40.028957 22818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:40.029009 22818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:40.029081 22818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:59:40.029132 22818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:59:40.038436 22818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:41047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:40.039623 22818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:40.041299 22818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:40.052443 22823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:40.052485 22826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:40.052510 22824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:40.053432 22818 server_base.cc:1034] running on GCE node
I20260928 02:59:40.054440 22818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:40.055769 22818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:40.056994 22818 hybrid_clock.cc:648] HybridClock initialized: now 1790564380056941 us; error 43 us; skew 500 ppm
I20260928 02:59:40.057313 22818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:59:40.059535 22818 webserver.cc:460] Webserver started at http://127.12.182.126:41781/ using document root <none> and password file <none>
I20260928 02:59:40.060369 22818 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:40.060480 22818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:40.060807 22818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:40.064798 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "cd59cfbadef140c9892df1201dff9afb"
format_stamp: "Formatted at 2026-09-28 02:59:40 on dist-test-slave-k5rg"
server_key: "718cb030452d4b9b91dedaec86b552d2"
server_key_iv: "be24056c2ab384ff2d6a622388718996"
server_key_version: "encryptionkey@0"
I20260928 02:59:40.065987 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "cd59cfbadef140c9892df1201dff9afb"
format_stamp: "Formatted at 2026-09-28 02:59:40 on dist-test-slave-k5rg"
server_key: "718cb030452d4b9b91dedaec86b552d2"
server_key_iv: "be24056c2ab384ff2d6a622388718996"
server_key_version: "encryptionkey@0"
I20260928 02:59:40.072726 22818 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 02:59:40.077095 22832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:40.078944 22818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:59:40.079110 22818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "cd59cfbadef140c9892df1201dff9afb"
format_stamp: "Formatted at 2026-09-28 02:59:40 on dist-test-slave-k5rg"
server_key: "718cb030452d4b9b91dedaec86b552d2"
server_key_iv: "be24056c2ab384ff2d6a622388718996"
server_key_version: "encryptionkey@0"
I20260928 02:59:40.079355 22818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:40.086491 22818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:40.087256 22818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:40.087574 22818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:40.106518 22818 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:41047
I20260928 02:59:40.106539 22883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:41047 every 8 connection(s)
I20260928 02:59:40.108217 22818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:59:40.111830 22884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:40.116885 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 22818
I20260928 02:59:40.117094 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:59:40.117718 13017 external_mini_cluster.cc:1428] Setting key 5ba69a1a6f0761b1bbf4f0c6ac9f78f8
I20260928 02:59:40.122216 22884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb: Bootstrap starting.
I20260928 02:59:40.125588 22884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:40.126663 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:40.119619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.12.182.126:41047)
0928 02:59:40.120620 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:40.120634 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:40.120674 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:59:40.122306 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:40.122327 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:40.122752 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:40.123028 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:40.124887 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:40.124896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:40.124915 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:40.124969 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:40.125049 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:40.125078 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:40.125258 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:40.125385 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:59:40.125683 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":352,"threads_started":1}
I20260928 02:59:40.126749 22884 log.cc:824] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:40.129513 22884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb: No bootstrap required, opened a new log
I20260928 02:59:40.136471 22884 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd59cfbadef140c9892df1201dff9afb" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41047 } }
I20260928 02:59:40.136777 22884 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:40.136845 22884 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59cfbadef140c9892df1201dff9afb, State: Initialized, Role: FOLLOWER
I20260928 02:59:40.137328 22884 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd59cfbadef140c9892df1201dff9afb" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41047 } }
I20260928 02:59:40.137542 22884 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:59:40.137666 22884 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:59:40.137791 22884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:40.139787 22884 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd59cfbadef140c9892df1201dff9afb" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41047 } }
I20260928 02:59:40.140306 22884 leader_election.cc:302] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd59cfbadef140c9892df1201dff9afb; no voters:
I20260928 02:59:40.140874 22884 leader_election.cc:288] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:59:40.140990 22889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:40.141664 22889 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [term 1 LEADER]: Becoming Leader. State: Replica: cd59cfbadef140c9892df1201dff9afb, State: Running, Role: LEADER
I20260928 02:59:40.142174 22889 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd59cfbadef140c9892df1201dff9afb" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41047 } }
I20260928 02:59:40.142771 22884 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:59:40.144701 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd59cfbadef140c9892df1201dff9afb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd59cfbadef140c9892df1201dff9afb" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41047 } } }
I20260928 02:59:40.145092 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:40.145658 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd59cfbadef140c9892df1201dff9afb. Latest consensus state: current_term: 1 leader_uuid: "cd59cfbadef140c9892df1201dff9afb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd59cfbadef140c9892df1201dff9afb" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41047 } } }
I20260928 02:59:40.145996 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:40.146548 22898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:59:40.151451 22898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:59:40.159420 22898 catalog_manager.cc:1347] Generated new cluster ID: 900ce2be626f4d97b2c8c8bfe7771951
I20260928 02:59:40.159513 22898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:59:40.187585 22898 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:59:40.187955 22898 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:59:40.196031 22898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd59cfbadef140c9892df1201dff9afb: Generated new TSK 0
I20260928 02:59:40.196877 22898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:59:40.213461 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41047
--builtin_ntp_servers=127.12.182.84:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:59:40.519812 22908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:40.520110 22908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:40.520237 22908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:40.527494 22908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:40.527604 22908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:40.527750 22908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:59:40.536783 22908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:40.538048 22908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:40.539711 22908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:40.552305 22914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:40.552322 22916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:40.552867 22913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:40.553133 22908 server_base.cc:1034] running on GCE node
I20260928 02:59:40.553678 22908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:40.554527 22908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:40.555711 22908 hybrid_clock.cc:648] HybridClock initialized: now 1790564380555671 us; error 32 us; skew 500 ppm
I20260928 02:59:40.555987 22908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:59:40.558100 22908 webserver.cc:460] Webserver started at http://127.12.182.65:33741/ using document root <none> and password file <none>
I20260928 02:59:40.558727 22908 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:40.558812 22908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:40.559118 22908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:40.561916 22908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "4c7d7010bed44aac94ac6b336e2974e6"
format_stamp: "Formatted at 2026-09-28 02:59:40 on dist-test-slave-k5rg"
server_key: "a2924237654409fef59307c35af49769"
server_key_iv: "06c00cc61f2e7f4b62236f3ccbc216d7"
server_key_version: "encryptionkey@0"
I20260928 02:59:40.562659 22908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "4c7d7010bed44aac94ac6b336e2974e6"
format_stamp: "Formatted at 2026-09-28 02:59:40 on dist-test-slave-k5rg"
server_key: "a2924237654409fef59307c35af49769"
server_key_iv: "06c00cc61f2e7f4b62236f3ccbc216d7"
server_key_version: "encryptionkey@0"
I20260928 02:59:40.567610 22908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:59:40.571004 22922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:40.572235 22908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:59:40.572381 22908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "4c7d7010bed44aac94ac6b336e2974e6"
format_stamp: "Formatted at 2026-09-28 02:59:40 on dist-test-slave-k5rg"
server_key: "a2924237654409fef59307c35af49769"
server_key_iv: "06c00cc61f2e7f4b62236f3ccbc216d7"
server_key_version: "encryptionkey@0"
I20260928 02:59:40.572588 22908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:40.577886 22908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:40.578613 22908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:40.578841 22908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:40.579663 22908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:40.580942 22908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:40.581043 22908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:40.581146 22908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:40.581212 22908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:40.609858 22908 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:45847
I20260928 02:59:40.609905 23034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:45847 every 8 connection(s)
I20260928 02:59:40.611063 22908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:59:40.612267 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 22908
I20260928 02:59:40.612470 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:59:40.613191 13017 external_mini_cluster.cc:1428] Setting key 88b8681d4f6e23d4dfb92de970debd43
I20260928 02:59:40.615933 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41047
--builtin_ntp_servers=127.12.182.84:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:59:40.618850 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:40.613717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:44079 (local address 127.12.182.126:41047)
0928 02:59:40.613915 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:40.613921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:40.615316 (+ 1395us) server_negotiation.cc:407] Connection header received
0928 02:59:40.616920 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:40.616925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:40.616988 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:40.617099 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:40.618254 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:40.618260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:40.618264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:40.618281 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:40.618319 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:40.618332 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:40.618447 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:40.618565 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 02:59:40.618622 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:59:40.619704 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:40.614207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41047 (local address 127.12.182.65:44079)
0928 02:59:40.615188 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:40.615228 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:40.616668 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:40.617266 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:40.617273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:40.617360 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:40.617985 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:59:40.617988 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:40.618022 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:59:40.618106 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:40.618413 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:40.618418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:40.618423 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:59:40.618606 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 02:59:40.618783 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":273,"threads_started":1}
I20260928 02:59:40.621026 23035 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41047
I20260928 02:59:40.621289 23035 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:40.621855 23035 heartbeater.cc:511] Master 127.12.182.126:41047 requested a full tablet report, sending...
I20260928 02:59:40.623726 22849 ts_manager.cc:194] Registered new tserver with Master: 4c7d7010bed44aac94ac6b336e2974e6 (127.12.182.65:45847)
W20260928 02:59:40.920207 23038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:40.920545 23038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:40.920674 23038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:40.930860 23038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:40.930954 23038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:40.931066 23038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:59:40.939689 23038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:40.940778 23038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:40.942414 23038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:40.955309 23043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:40.955363 23046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:40.955339 23044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:40.956321 23038 server_base.cc:1034] running on GCE node
I20260928 02:59:40.956822 23038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:40.957646 23038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:40.958842 23038 hybrid_clock.cc:648] HybridClock initialized: now 1790564380958802 us; error 34 us; skew 500 ppm
I20260928 02:59:40.959151 23038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:59:40.961087 23038 webserver.cc:460] Webserver started at http://127.12.182.66:45647/ using document root <none> and password file <none>
I20260928 02:59:40.961733 23038 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:40.961823 23038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:40.962111 23038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:40.966377 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "9e07f628ad164650856f33a5b0aa0f82"
format_stamp: "Formatted at 2026-09-28 02:59:40 on dist-test-slave-k5rg"
server_key: "4499044a7157710cb6a8f6009ab90540"
server_key_iv: "d23061aaf1f16114849f077de4bf9228"
server_key_version: "encryptionkey@0"
I20260928 02:59:40.967108 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "9e07f628ad164650856f33a5b0aa0f82"
format_stamp: "Formatted at 2026-09-28 02:59:40 on dist-test-slave-k5rg"
server_key: "4499044a7157710cb6a8f6009ab90540"
server_key_iv: "d23061aaf1f16114849f077de4bf9228"
server_key_version: "encryptionkey@0"
I20260928 02:59:40.971697 23038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:59:40.974854 23052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:40.976107 23038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:59:40.976224 23038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "9e07f628ad164650856f33a5b0aa0f82"
format_stamp: "Formatted at 2026-09-28 02:59:40 on dist-test-slave-k5rg"
server_key: "4499044a7157710cb6a8f6009ab90540"
server_key_iv: "d23061aaf1f16114849f077de4bf9228"
server_key_version: "encryptionkey@0"
I20260928 02:59:40.976397 23038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:40.981545 23038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:40.982259 23038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:40.982461 23038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:40.983237 23038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:40.984530 23038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:40.984609 23038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:40.984684 23038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:40.984737 23038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:41.013191 23038 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:37701
I20260928 02:59:41.013234 23164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:37701 every 8 connection(s)
I20260928 02:59:41.014434 23038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:59:41.015218 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 23038
I20260928 02:59:41.015390 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:59:41.015931 13017 external_mini_cluster.cc:1428] Setting key 6eb32e605b7d5b269c82dc2ab0932f6a
I20260928 02:59:41.018894 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41047
--builtin_ntp_servers=127.12.182.84:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:59:41.022913 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:41.017062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:34413 (local address 127.12.182.126:41047)
0928 02:59:41.017243 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:41.017248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:41.018769 (+ 1521us) server_negotiation.cc:407] Connection header received
0928 02:59:41.020917 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:41.020920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:41.020978 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:41.021058 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:41.022340 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:41.022344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:41.022348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:41.022364 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:41.022395 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:41.022406 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:41.022505 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:41.022664 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:59:41.022711 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:59:41.024000 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:41.017541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41047 (local address 127.12.182.66:34413)
0928 02:59:41.018609 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:41.018660 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:41.020653 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:41.021282 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:41.021291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:41.021399 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:41.022035 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:59:41.022038 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:41.022083 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:59:41.022183 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:41.022488 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:41.022497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:41.022503 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:59:41.022735 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 02:59:41.022966 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":305,"threads_started":1}
I20260928 02:59:41.025177 23165 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41047
I20260928 02:59:41.025450 23165 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:41.026082 23165 heartbeater.cc:511] Master 127.12.182.126:41047 requested a full tablet report, sending...
I20260928 02:59:41.027449 22849 ts_manager.cc:194] Registered new tserver with Master: 9e07f628ad164650856f33a5b0aa0f82 (127.12.182.66:37701)
W20260928 02:59:41.320194 23168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:41.320446 23168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:41.320572 23168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:41.327531 23168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:41.327629 23168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:41.327770 23168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:59:41.339248 23168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:41.340814 23168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:41.342896 23168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:41.356101 23173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:41.356156 23174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:41.356278 23176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:41.357085 23168 server_base.cc:1034] running on GCE node
I20260928 02:59:41.357631 23168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:41.358470 23168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:41.359652 23168 hybrid_clock.cc:648] HybridClock initialized: now 1790564381359611 us; error 32 us; skew 500 ppm
I20260928 02:59:41.359925 23168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:59:41.362051 23168 webserver.cc:460] Webserver started at http://127.12.182.67:39471/ using document root <none> and password file <none>
I20260928 02:59:41.362654 23168 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:41.362746 23168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:41.363026 23168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:41.365655 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "a588397a94ae4029bbb5a7cd0e6706dc"
format_stamp: "Formatted at 2026-09-28 02:59:41 on dist-test-slave-k5rg"
server_key: "f5ff0a1f5e7347c99222555365d9b896"
server_key_iv: "7e04cf1f4c48047791b823cc41a52c2b"
server_key_version: "encryptionkey@0"
I20260928 02:59:41.366379 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "a588397a94ae4029bbb5a7cd0e6706dc"
format_stamp: "Formatted at 2026-09-28 02:59:41 on dist-test-slave-k5rg"
server_key: "f5ff0a1f5e7347c99222555365d9b896"
server_key_iv: "7e04cf1f4c48047791b823cc41a52c2b"
server_key_version: "encryptionkey@0"
I20260928 02:59:41.370982 23168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:59:41.374037 23182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:41.375242 23168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:59:41.375378 23168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "a588397a94ae4029bbb5a7cd0e6706dc"
format_stamp: "Formatted at 2026-09-28 02:59:41 on dist-test-slave-k5rg"
server_key: "f5ff0a1f5e7347c99222555365d9b896"
server_key_iv: "7e04cf1f4c48047791b823cc41a52c2b"
server_key_version: "encryptionkey@0"
I20260928 02:59:41.375557 23168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:41.381124 23168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:41.382022 23168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:41.382298 23168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:41.383266 23168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:41.384611 23168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:41.384691 23168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:41.384768 23168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:41.384819 23168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:41.413223 23168 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:37835
I20260928 02:59:41.413272 23294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:37835 every 8 connection(s)
I20260928 02:59:41.414469 23168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:59:41.419013 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 23168
I20260928 02:59:41.419166 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:59:41.419565 13017 external_mini_cluster.cc:1428] Setting key dfd5203574596de3b8087f794ff392bc
I20260928 02:59:41.421954 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:41.416770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:40719 (local address 127.12.182.126:41047)
0928 02:59:41.416945 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:41.416950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:41.418221 (+ 1271us) server_negotiation.cc:407] Connection header received
0928 02:59:41.419864 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:41.419869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:41.419936 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:41.420023 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:41.421265 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:41.421271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:41.421275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:41.421294 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:41.421324 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:41.421335 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:41.421426 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:41.421637 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:59:41.421696 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:59:41.422771 23297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:41.417223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41047 (local address 127.12.182.67:40719)
0928 02:59:41.418067 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:41.418108 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:41.419612 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:41.420246 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:41.420254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:41.420339 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:41.420984 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:59:41.420986 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:41.421022 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:59:41.421111 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:41.421418 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:41.421426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:41.421434 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:59:41.421687 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 02:59:41.421916 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":240,"threads_started":1}
I20260928 02:59:41.423856 23295 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41047
I20260928 02:59:41.424111 23295 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:41.424626 23295 heartbeater.cc:511] Master 127.12.182.126:41047 requested a full tablet report, sending...
I20260928 02:59:41.425848 22849 ts_manager.cc:194] Registered new tserver with Master: a588397a94ae4029bbb5a7cd0e6706dc (127.12.182.67:37835)
I20260928 02:59:41.434301 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:59:41.439893 22887 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:59:41.438612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.12.182.126:41047)
0928 02:59:41.438759 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:41.438763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:41.439195 (+ 432us) server_negotiation.cc:407] Connection header received
0928 02:59:41.439322 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:41.439326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:41.439373 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:41.439462 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:41.439698 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60400: BlockingRecv error: recv got EOF from 127.0.0.1:60400 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:59:41.440423 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 22908
I20260928 02:59:41.443531 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:41.441240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.12.182.65:45847)
0928 02:59:41.441733 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:41.441739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:41.441764 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:59:41.441819 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:41.441825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:41.442017 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:41.442125 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:41.442904 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:41.442910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:41.442920 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:41.442932 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:41.442970 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:41.442986 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:41.443109 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:41.443172 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:59:41.443265 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":206,"threads_started":1}
I20260928 02:59:41.444626 23030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:41.522778 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 22908
I20260928 02:59:41.535630 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 23038
I20260928 02:59:41.538757 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:41.536451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37608 (local address 127.12.182.66:37701)
0928 02:59:41.536937 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:41.536944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:41.536969 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:59:41.537028 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:41.537034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:41.537245 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:41.537363 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:41.538223 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:41.538229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:41.538241 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:41.538257 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:41.538295 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:41.538308 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:41.538405 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:41.538464 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:59:41.538539 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":229,"threads_started":1}
I20260928 02:59:41.539870 23160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:41.618634 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 23038
I20260928 02:59:41.630887 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 23168
I20260928 02:59:41.634642 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:41.631850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.12.182.67:37835)
0928 02:59:41.632539 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:41.632546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:41.632576 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:59:41.632636 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:41.632643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:41.632852 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:41.633024 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:41.634003 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:41.634009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:41.634021 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:59:41.634037 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:41.634076 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:59:41.634090 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:41.634214 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:41.634260 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 02:59:41.634334 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":372,"threads_started":1}
I20260928 02:59:41.635744 23290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:41.715974 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 23168
I20260928 02:59:41.728454 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 22818
I20260928 02:59:41.729537 22879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:41.828054 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 22818
2026-09-28T02:59:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2169 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:59:41 dist-test-slave-k5rg krb5kdc[23313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:59:41 dist-test-slave-k5rg krb5kdc[23313](info): set up 2 sockets
Sep 28 02:59:41 dist-test-slave-k5rg krb5kdc[23313](info): commencing operation
krb5kdc: starting...
W20260928 02:59:42.886369 13017 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:59:43 dist-test-slave-k5rg krb5kdc[23313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 02:59:43.090994 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:35925
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:43.394950 23326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:43.395197 23326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:43.395325 23326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:43.402292 23326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:59:43.402388 23326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:43.402462 23326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:43.402530 23326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:59:43.402587 23326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:59:43.411877 23326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:35925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:43.413040 23326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:43.414268 23326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:43.424783 23331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:43.424841 23332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:43.424937 23334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:43.425927 23326 server_base.cc:1034] running on GCE node
I20260928 02:59:43.426609 23326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:43.427846 23326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:43.429060 23326 hybrid_clock.cc:648] HybridClock initialized: now 1790564383429007 us; error 34 us; skew 500 ppm
I20260928 02:59:43.429379 23326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:43 dist-test-slave-k5rg krb5kdc[23313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564383, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:43.432130 23326 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 02:59:43.433404 23326 webserver.cc:460] Webserver started at http://127.12.182.126:35301/ using document root <none> and password file <none>
I20260928 02:59:43.434063 23326 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:43.434180 23326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:43.434445 23326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:43.437163 23326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "7bb391031daa46b4a3edff866483fddd"
format_stamp: "Formatted at 2026-09-28 02:59:43 on dist-test-slave-k5rg"
server_key: "8b9ef7a365923b91b73727c206d94793"
server_key_iv: "173421ab285297d6a30cb60c7a1d2c6d"
server_key_version: "encryptionkey@0"
I20260928 02:59:43.437901 23326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "7bb391031daa46b4a3edff866483fddd"
format_stamp: "Formatted at 2026-09-28 02:59:43 on dist-test-slave-k5rg"
server_key: "8b9ef7a365923b91b73727c206d94793"
server_key_iv: "173421ab285297d6a30cb60c7a1d2c6d"
server_key_version: "encryptionkey@0"
I20260928 02:59:43.442667 23326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:59:43.445834 23341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:43.447031 23326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:59:43.447177 23326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "7bb391031daa46b4a3edff866483fddd"
format_stamp: "Formatted at 2026-09-28 02:59:43 on dist-test-slave-k5rg"
server_key: "8b9ef7a365923b91b73727c206d94793"
server_key_iv: "173421ab285297d6a30cb60c7a1d2c6d"
server_key_version: "encryptionkey@0"
I20260928 02:59:43.447342 23326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:43.462123 23326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:43.465711 23326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:43.465948 23326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:43.482069 23326 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:35925
I20260928 02:59:43.482087 23392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:35925 every 8 connection(s)
I20260928 02:59:43.483280 23326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 02:59:43.486696 23393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:43.488054 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 23326
I20260928 02:59:43.488272 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 02:59:43.489058 13017 external_mini_cluster.cc:1428] Setting key a1b4dd894fb811bb9d1d0de82cf36db9
I20260928 02:59:43.500211 23393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd: Bootstrap starting.
Sep 28 02:59:43 dist-test-slave-k5rg krb5kdc[23313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:59:43.503844 23393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:43.504972 23393 log.cc:824] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:43.507894 23393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd: No bootstrap required, opened a new log
I20260928 02:59:43.510735 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:43.490933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.12.182.126:35925)
0928 02:59:43.491658 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:43.491668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:43.491715 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:59:43.493208 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:43.493228 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:43.493597 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:43.493940 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:43.495619 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:43.496861 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:43.498628 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:43.499270 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:43.499393 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:43.503855 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:43.503863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:43.503876 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:43.503918 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:43.507029 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:43.507754 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:43.507759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:43.507765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:43.507916 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:43.508291 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:43.508294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:43.508296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:43.508778 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:43.509001 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:43.509385 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 02:59:43.509763 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":532,"thread_start_us":317,"threads_started":1}
I20260928 02:59:43.515416 23393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb391031daa46b4a3edff866483fddd" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35925 } }
I20260928 02:59:43.515718 23393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:43.515810 23393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb391031daa46b4a3edff866483fddd, State: Initialized, Role: FOLLOWER
I20260928 02:59:43.516300 23393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb391031daa46b4a3edff866483fddd" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35925 } }
I20260928 02:59:43.516494 23393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:59:43.516615 23393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:59:43.516746 23393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:43.518775 23393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb391031daa46b4a3edff866483fddd" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35925 } }
I20260928 02:59:43.519196 23393 leader_election.cc:302] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bb391031daa46b4a3edff866483fddd; no voters:
I20260928 02:59:43.519697 23393 leader_election.cc:288] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:59:43.519799 23398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:43.520421 23398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [term 1 LEADER]: Becoming Leader. State: Replica: 7bb391031daa46b4a3edff866483fddd, State: Running, Role: LEADER
I20260928 02:59:43.521044 23398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb391031daa46b4a3edff866483fddd" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35925 } }
I20260928 02:59:43.521780 23393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:59:43.523728 23399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bb391031daa46b4a3edff866483fddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb391031daa46b4a3edff866483fddd" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35925 } } }
I20260928 02:59:43.524087 23399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:43.524070 23400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb391031daa46b4a3edff866483fddd. Latest consensus state: current_term: 1 leader_uuid: "7bb391031daa46b4a3edff866483fddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb391031daa46b4a3edff866483fddd" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35925 } } }
I20260928 02:59:43.525367 23400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd [sys.catalog]: This master's current role is: LEADER
I20260928 02:59:43.525631 23407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:59:43.530404 23407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:59:43.539180 23407 catalog_manager.cc:1347] Generated new cluster ID: 5cd6947b4aa74f7c9228eb0d11dd31ff
I20260928 02:59:43.539275 23407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:59:43.553505 23407 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:59:43.554661 23407 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:59:43.563048 23407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd: Generated new TSK 0
I20260928 02:59:43.563772 23407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 02:59:43.619339 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:35925
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:43.921348 23419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:43.921671 23419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:43.921815 23419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:43.929047 23419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:43.929162 23419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:43.929312 23419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 02:59:43.938566 23419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:43.939731 23419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:43.940963 23419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:43.953410 23427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:43.953429 23424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:43.953429 23425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:43.954007 23419 server_base.cc:1034] running on GCE node
I20260928 02:59:43.954542 23419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:43.955351 23419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:43.956557 23419 hybrid_clock.cc:648] HybridClock initialized: now 1790564383956498 us; error 45 us; skew 500 ppm
I20260928 02:59:43.956842 23419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:43 dist-test-slave-k5rg krb5kdc[23313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564383, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:43.963668 23419 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 02:59:43.964929 23419 webserver.cc:460] Webserver started at http://127.12.182.65:39529/ using document root <none> and password file <none>
I20260928 02:59:43.965526 23419 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:43.965648 23419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:43.965967 23419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:43.968588 23419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "59bb16d1c39c443fa6118e0173b933ed"
format_stamp: "Formatted at 2026-09-28 02:59:43 on dist-test-slave-k5rg"
server_key: "8da659f8746685e2aacbac163e601ce2"
server_key_iv: "04db386417b0d6f2ea25367db071fb2a"
server_key_version: "encryptionkey@0"
I20260928 02:59:43.969287 23419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "59bb16d1c39c443fa6118e0173b933ed"
format_stamp: "Formatted at 2026-09-28 02:59:43 on dist-test-slave-k5rg"
server_key: "8da659f8746685e2aacbac163e601ce2"
server_key_iv: "04db386417b0d6f2ea25367db071fb2a"
server_key_version: "encryptionkey@0"
I20260928 02:59:43.973976 23419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:59:43.977137 23434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:43.978331 23419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:59:43.978473 23419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "59bb16d1c39c443fa6118e0173b933ed"
format_stamp: "Formatted at 2026-09-28 02:59:43 on dist-test-slave-k5rg"
server_key: "8da659f8746685e2aacbac163e601ce2"
server_key_iv: "04db386417b0d6f2ea25367db071fb2a"
server_key_version: "encryptionkey@0"
I20260928 02:59:43.978631 23419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:44.019589 23419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:44.023700 23419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:44.023993 23419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:44.024969 23419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:44.026325 23419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:44.026398 23419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:44.026485 23419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:44.026526 23419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:44.059818 23419 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:42583
I20260928 02:59:44.059862 23546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:42583 every 8 connection(s)
I20260928 02:59:44.061029 23419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 02:59:44.069281 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 23419
I20260928 02:59:44.069473 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 02:59:44.070118 13017 external_mini_cluster.cc:1428] Setting key a78c73d25e4cafc880e1863c144a36c8
Sep 28 02:59:44 dist-test-slave-k5rg krb5kdc[23313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564383, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:59:44.079963 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:44.063493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:36035 (local address 127.12.182.126:35925)
0928 02:59:44.063921 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:44.063925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:44.064907 (+ 982us) server_negotiation.cc:407] Connection header received
0928 02:59:44.066651 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:44.066656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:44.066718 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:44.066810 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:44.068152 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:44.068847 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:44.070471 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:44.070940 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:44.071055 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:44.074396 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:44.074401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:44.074405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:44.074440 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:44.077250 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:44.077956 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:44.077961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:44.077964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:44.078096 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:44.078459 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:44.078464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:44.078466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:44.078766 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:44.078910 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:44.079557 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 02:59:44.079721 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":197,"threads_started":1}
I20260928 02:59:44.081248 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:44.064003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:35925 (local address 127.12.182.65:36035)
0928 02:59:44.064699 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:44.064771 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:44.066416 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:44.066987 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:44.066995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:44.067489 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:44.068018 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:44.068044 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:44.068996 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:44.069000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:44.070337 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:44.070348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:44.071164 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:44.071172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:44.071383 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:44.071979 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:44.072001 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:44.074202 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:44.077376 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:44.077382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:44.077394 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:44.077848 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:44.078190 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:44.078193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:44.078196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:44.078363 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:44.078861 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:44.078866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:44.079173 (+ 307us) client_negotiation.cc:769] Sending connection context
0928 02:59:44.079452 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 02:59:44.079952 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":274,"threads_started":1}
I20260928 02:59:44.082765 23547 heartbeater.cc:347] Connected to a master server at 127.12.182.126:35925
I20260928 02:59:44.083040 23547 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:44.083735 23547 heartbeater.cc:511] Master 127.12.182.126:35925 requested a full tablet report, sending...
I20260928 02:59:44.085632 23358 ts_manager.cc:194] Registered new tserver with Master: 59bb16d1c39c443fa6118e0173b933ed (127.12.182.65:42583)
I20260928 02:59:44.087014 23358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:36035
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 02:59:44.117146 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:35925
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:44.417905 23553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:44.418154 23553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:44.418272 23553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:44.425383 23553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:44.425485 23553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:44.425657 23553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 02:59:44.435077 23553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:44.436335 23553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:44.437574 23553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:44.449774 23561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:44.449836 23559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:44.450003 23558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:44.451679 23553 server_base.cc:1034] running on GCE node
I20260928 02:59:44.452227 23553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:44.453056 23553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:44.454237 23553 hybrid_clock.cc:648] HybridClock initialized: now 1790564384454200 us; error 31 us; skew 500 ppm
I20260928 02:59:44.454496 23553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:44 dist-test-slave-k5rg krb5kdc[23313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564384, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:44.457312 23553 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 02:59:44.458621 23553 webserver.cc:460] Webserver started at http://127.12.182.66:37473/ using document root <none> and password file <none>
I20260928 02:59:44.459249 23553 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:44.459342 23553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:44.459574 23553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:44.462312 23553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "8281691b0f274f3cbbd3d129a4e08fdc"
format_stamp: "Formatted at 2026-09-28 02:59:44 on dist-test-slave-k5rg"
server_key: "b5913d487c2903d8b2aae9e2bdbcff57"
server_key_iv: "bac97eef2c82864a2e8318a6f11ad0c3"
server_key_version: "encryptionkey@0"
I20260928 02:59:44.462970 23553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "8281691b0f274f3cbbd3d129a4e08fdc"
format_stamp: "Formatted at 2026-09-28 02:59:44 on dist-test-slave-k5rg"
server_key: "b5913d487c2903d8b2aae9e2bdbcff57"
server_key_iv: "bac97eef2c82864a2e8318a6f11ad0c3"
server_key_version: "encryptionkey@0"
I20260928 02:59:44.467658 23553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:59:44.470811 23568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:44.472062 23553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:59:44.472198 23553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "8281691b0f274f3cbbd3d129a4e08fdc"
format_stamp: "Formatted at 2026-09-28 02:59:44 on dist-test-slave-k5rg"
server_key: "b5913d487c2903d8b2aae9e2bdbcff57"
server_key_iv: "bac97eef2c82864a2e8318a6f11ad0c3"
server_key_version: "encryptionkey@0"
I20260928 02:59:44.472359 23553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:44.490922 23553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:44.495484 23553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:44.495762 23553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:44.496797 23553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:44.498349 23553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:44.498431 23553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:44.498538 23553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:44.498591 23553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:44.527663 23553 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:36693
I20260928 02:59:44.527695 23680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:36693 every 8 connection(s)
I20260928 02:59:44.528877 23553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 02:59:44.536131 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 23553
I20260928 02:59:44.536324 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 02:59:44.536803 13017 external_mini_cluster.cc:1428] Setting key 9fbb1762560329f29880c3c89796d57d
Sep 28 02:59:44 dist-test-slave-k5rg krb5kdc[23313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564384, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:59:44.547402 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:44.531157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:50315 (local address 127.12.182.126:35925)
0928 02:59:44.531365 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:44.531371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:44.532418 (+ 1047us) server_negotiation.cc:407] Connection header received
0928 02:59:44.534146 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:44.534150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:44.534216 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:44.534292 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:44.535584 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:44.536542 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:44.538197 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:44.538713 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:44.538843 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:44.542197 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:44.542202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:44.542205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:44.542237 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:44.544604 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:44.545264 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:44.545267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:44.545269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:44.545388 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:44.545826 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:44.545830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:44.545832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:44.546098 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:44.546224 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:44.546859 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 02:59:44.547073 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:59:44.548326 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:44.531621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:35925 (local address 127.12.182.66:50315)
0928 02:59:44.532276 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:44.532311 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:44.533898 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:44.534470 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:44.534478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:44.534932 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:44.535460 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:44.535476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:44.536671 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:44.536676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:44.538051 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:44.538064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:44.538936 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:44.538943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:44.539156 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:44.539796 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:44.539822 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:44.541972 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:44.544730 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:44.544738 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:44.544755 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:44.545145 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:44.545487 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:44.545492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:44.545496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:44.545721 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:44.546214 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:44.546221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:44.546531 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 02:59:44.546754 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 02:59:44.547255 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":251,"threads_started":1}
I20260928 02:59:44.549645 23681 heartbeater.cc:347] Connected to a master server at 127.12.182.126:35925
I20260928 02:59:44.549916 23681 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:44.550659 23681 heartbeater.cc:511] Master 127.12.182.126:35925 requested a full tablet report, sending...
I20260928 02:59:44.551961 23358 ts_manager.cc:194] Registered new tserver with Master: 8281691b0f274f3cbbd3d129a4e08fdc (127.12.182.66:36693)
I20260928 02:59:44.552724 23358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:50315
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 02:59:44.582511 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:35925
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:59:44.885011 23686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:59:44.885241 23686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:59:44.885356 23686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:59:44.892251 23686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:59:44.892341 23686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:59:44.892491 23686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 02:59:44.901518 23686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:35925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:59:44.902637 23686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:59:44.903860 23686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:59:44.915561 23691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:44.915561 23692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:59:44.915709 23694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:59:44.916344 23686 server_base.cc:1034] running on GCE node
I20260928 02:59:44.916882 23686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:59:44.917793 23686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:59:44.918980 23686 hybrid_clock.cc:648] HybridClock initialized: now 1790564384918929 us; error 45 us; skew 500 ppm
I20260928 02:59:44.919269 23686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:59:44 dist-test-slave-k5rg krb5kdc[23313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564384, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:59:44.922099 23686 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 02:59:44.923774 23686 webserver.cc:460] Webserver started at http://127.12.182.67:45759/ using document root <none> and password file <none>
I20260928 02:59:44.924410 23686 fs_manager.cc:359] Metadata directory not provided
I20260928 02:59:44.924518 23686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:59:44.924794 23686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:59:44.927536 23686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "d0b4dacd1fc84296a001c5992a5b1bb9"
format_stamp: "Formatted at 2026-09-28 02:59:44 on dist-test-slave-k5rg"
server_key: "ab5a0c0a525c7df7c096494d2d025e83"
server_key_iv: "af19f979536ef1ed03e67d9cb968c8eb"
server_key_version: "encryptionkey@0"
I20260928 02:59:44.928196 23686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "d0b4dacd1fc84296a001c5992a5b1bb9"
format_stamp: "Formatted at 2026-09-28 02:59:44 on dist-test-slave-k5rg"
server_key: "ab5a0c0a525c7df7c096494d2d025e83"
server_key_iv: "af19f979536ef1ed03e67d9cb968c8eb"
server_key_version: "encryptionkey@0"
I20260928 02:59:44.932829 23686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:59:44.935948 23701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:44.937165 23686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:59:44.937306 23686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "d0b4dacd1fc84296a001c5992a5b1bb9"
format_stamp: "Formatted at 2026-09-28 02:59:44 on dist-test-slave-k5rg"
server_key: "ab5a0c0a525c7df7c096494d2d025e83"
server_key_iv: "af19f979536ef1ed03e67d9cb968c8eb"
server_key_version: "encryptionkey@0"
I20260928 02:59:44.937464 23686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:59:44.969869 23686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:59:44.973570 23686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:59:44.973855 23686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:59:44.974722 23686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:59:44.975961 23686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:59:44.976046 23686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:44.976166 23686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:59:44.976215 23686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:59:45.004737 23686 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:38467
I20260928 02:59:45.004772 23813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:38467 every 8 connection(s)
I20260928 02:59:45.005951 23686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 02:59:45.010228 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 23686
I20260928 02:59:45.010389 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 02:59:45.010771 13017 external_mini_cluster.cc:1428] Setting key 81702620787657ddeabc6367072874a9
Sep 28 02:59:45 dist-test-slave-k5rg krb5kdc[23313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564384, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 02:59:45.023747 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.008229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:49247 (local address 127.12.182.126:35925)
0928 02:59:45.008400 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.008404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.009463 (+ 1059us) server_negotiation.cc:407] Connection header received
0928 02:59:45.011274 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.011280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.011363 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.011464 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:45.013174 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.014002 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.015509 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.015897 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.015988 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.019054 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:45.019059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:45.019062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:45.019093 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:45.021257 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:45.021916 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:45.021920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:45.021922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:45.022032 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:45.022395 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:45.022398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:45.022400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:45.022643 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:45.022781 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.023347 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.023506 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:59:45.024751 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.008666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:35925 (local address 127.12.182.67:49247)
0928 02:59:45.009307 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:45.009351 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:45.011018 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:45.011701 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:45.011712 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:45.012272 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:59:45.013010 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.013034 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.014162 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.014166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.015389 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.015401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.016062 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.016067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.016251 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.016812 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:59:45.016836 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:59:45.018883 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:59:45.021366 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:45.021371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:45.021382 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:45.021833 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:45.022139 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:59:45.022143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:59:45.022145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:59:45.022304 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:59:45.022760 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:59:45.022768 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:59:45.023059 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 02:59:45.023279 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 02:59:45.023615 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":256,"threads_started":1}
I20260928 02:59:45.026019 23814 heartbeater.cc:347] Connected to a master server at 127.12.182.126:35925
I20260928 02:59:45.026309 23814 heartbeater.cc:464] Registering TS with master...
I20260928 02:59:45.026953 23814 heartbeater.cc:511] Master 127.12.182.126:35925 requested a full tablet report, sending...
I20260928 02:59:45.028239 23357 ts_manager.cc:194] Registered new tserver with Master: d0b4dacd1fc84296a001c5992a5b1bb9 (127.12.182.67:38467)
I20260928 02:59:45.029038 23357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:49247
I20260928 02:59:45.037632 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:59:45.052908 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.041810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.12.182.126:35925)
0928 02:59:45.041956 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.041960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.042403 (+ 443us) server_negotiation.cc:407] Connection header received
0928 02:59:45.042548 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.042551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.042621 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.042699 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:45.043907 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.044682 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.045940 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.046332 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.046421 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.048336 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:45.048340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:45.048343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:45.048372 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:45.050783 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:45.051387 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:45.051392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:45.051394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:45.051531 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:45.051821 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:45.051823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:45.051825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:45.052104 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:45.052283 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.052493 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.052635 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:59:45.056550 23357 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:59:45.065035 23357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:59:45.089928 23547 heartbeater.cc:503] Master 127.12.182.126:35925 was elected leader, sending a full tablet report...
I20260928 02:59:45.099062 23827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.076734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:36693 (local address 127.0.0.1:51860)
0928 02:59:45.089418 (+ 12684us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:45.089436 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:45.089550 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:45.090587 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:45.090596 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:45.090629 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:45.091095 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.091109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.093141 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.093147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.095751 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.095767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.098042 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.098048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.098264 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.098314 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 02:59:45.098440 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:59:45.098640 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12556,"thread_start_us":12415,"threads_started":1}
I20260928 02:59:45.099977 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.076946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.12.182.66:36693)
0928 02:59:45.089060 (+ 12114us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.089070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.089533 (+ 463us) server_negotiation.cc:407] Connection header received
0928 02:59:45.089750 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.089759 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.090098 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.090273 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:45.091281 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.092938 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.095954 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.097486 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.097599 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.097830 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.099468 (+ 1638us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.099659 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11933,"thread_start_us":11821,"threads_started":1}
I20260928 02:59:45.102357 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.076439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:42583 (local address 127.0.0.1:59638)
0928 02:59:45.087970 (+ 11531us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:45.088028 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:45.088219 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:45.093539 (+ 5320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:45.093544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:45.093571 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:45.094114 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.094125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.096191 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.096198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.099520 (+ 3322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.099536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.101627 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.101634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.101793 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.101822 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:59:45.101903 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:59:45.102026 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11227,"thread_start_us":11507,"threads_started":1}
I20260928 02:59:45.102460 23828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.076698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.12.182.65:42583)
0928 02:59:45.089555 (+ 12857us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.089563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.089588 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:59:45.091740 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.091750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.093077 (+ 1327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.093266 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:45.094357 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.096020 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.099700 (+ 3680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.101193 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.101292 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.101521 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.101984 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.102146 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12685,"thread_start_us":12526,"threads_started":1}
I20260928 02:59:45.104704 23616 tablet_service.cc:1467] Processing CreateTablet for tablet 0126d76a517447ec8f4d0d52279fbe64 (DEFAULT_TABLE table=test-table [id=c70e70f66644439ca9ac6e102a61bd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:45.106258 23616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0126d76a517447ec8f4d0d52279fbe64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:45.106112 23482 tablet_service.cc:1467] Processing CreateTablet for tablet 0126d76a517447ec8f4d0d52279fbe64 (DEFAULT_TABLE table=test-table [id=c70e70f66644439ca9ac6e102a61bd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:45.107300 23827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.089764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:38467 (local address 127.0.0.1:37168)
0928 02:59:45.100562 (+ 10798us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:45.100581 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:45.100677 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:45.102097 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:45.102101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:45.102129 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:45.102283 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.102293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.104204 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.104207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.105642 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.105655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.106831 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.106835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.106983 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.107005 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:59:45.107051 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:59:45.107136 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10698,"thread_start_us":11123,"threads_started":1}
I20260928 02:59:45.107460 23482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0126d76a517447ec8f4d0d52279fbe64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:45.108091 23831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.089806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.12.182.67:38467)
0928 02:59:45.101497 (+ 11691us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.101506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.101536 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:59:45.101663 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.101671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.101899 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.102463 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:45.102730 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.104068 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.105785 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.106674 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.107583 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.107740 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.107792 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.107889 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11541,"thread_start_us":11420,"threads_started":1}
I20260928 02:59:45.110805 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 0126d76a517447ec8f4d0d52279fbe64 (DEFAULT_TABLE table=test-table [id=c70e70f66644439ca9ac6e102a61bd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:59:45.111791 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0126d76a517447ec8f4d0d52279fbe64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:59:45.120201 23832 tablet_bootstrap.cc:492] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: Bootstrap starting.
I20260928 02:59:45.122025 23833 tablet_bootstrap.cc:492] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: Bootstrap starting.
I20260928 02:59:45.123435 23834 tablet_bootstrap.cc:492] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: Bootstrap starting.
I20260928 02:59:45.124539 23832 tablet_bootstrap.cc:654] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:45.125267 23833 tablet_bootstrap.cc:654] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:45.126019 23832 log.cc:824] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:45.126470 23833 log.cc:824] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:45.127637 23834 tablet_bootstrap.cc:654] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: Neither blocks nor log segments found. Creating new log.
I20260928 02:59:45.128404 23832 tablet_bootstrap.cc:492] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: No bootstrap required, opened a new log
I20260928 02:59:45.128634 23832 ts_tablet_manager.cc:1397] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260928 02:59:45.128829 23834 log.cc:824] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:59:45.131671 23833 tablet_bootstrap.cc:492] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: No bootstrap required, opened a new log
I20260928 02:59:45.131670 23834 tablet_bootstrap.cc:492] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: No bootstrap required, opened a new log
I20260928 02:59:45.131920 23833 ts_tablet_manager.cc:1397] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260928 02:59:45.131920 23834 ts_tablet_manager.cc:1397] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 02:59:45.135479 23832 raft_consensus.cc:348] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } }
I20260928 02:59:45.135809 23832 raft_consensus.cc:374] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:45.135932 23832 raft_consensus.cc:729] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8281691b0f274f3cbbd3d129a4e08fdc, State: Initialized, Role: FOLLOWER
I20260928 02:59:45.136497 23832 consensus_queue.cc:260] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } }
I20260928 02:59:45.139210 23681 heartbeater.cc:503] Master 127.12.182.126:35925 was elected leader, sending a full tablet report...
I20260928 02:59:45.139909 23832 ts_tablet_manager.cc:1428] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260928 02:59:45.142055 23833 raft_consensus.cc:348] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } }
I20260928 02:59:45.142035 23834 raft_consensus.cc:348] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } }
I20260928 02:59:45.142448 23833 raft_consensus.cc:374] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:45.142453 23834 raft_consensus.cc:374] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:59:45.142547 23833 raft_consensus.cc:729] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59bb16d1c39c443fa6118e0173b933ed, State: Initialized, Role: FOLLOWER
I20260928 02:59:45.142549 23834 raft_consensus.cc:729] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b4dacd1fc84296a001c5992a5b1bb9, State: Initialized, Role: FOLLOWER
I20260928 02:59:45.143229 23834 consensus_queue.cc:260] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } }
I20260928 02:59:45.143229 23833 consensus_queue.cc:260] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } }
I20260928 02:59:45.144912 23814 heartbeater.cc:503] Master 127.12.182.126:35925 was elected leader, sending a full tablet report...
I20260928 02:59:45.145237 23834 ts_tablet_manager.cc:1428] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 02:59:45.145808 23833 ts_tablet_manager.cc:1428] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260928 02:59:45.257381 23815 tablet.cc:2369] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:59:45.280647 23682 tablet.cc:2369] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:59:45.313273 23548 tablet.cc:2369] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:59:45.427968 23840 raft_consensus.cc:482] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:59:45.428236 23840 raft_consensus.cc:504] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } }
I20260928 02:59:45.429886 23840 leader_election.cc:288] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8281691b0f274f3cbbd3d129a4e08fdc (127.12.182.66:36693), d0b4dacd1fc84296a001c5992a5b1bb9 (127.12.182.67:38467)
I20260928 02:59:45.436362 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.430315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:54181 (local address 127.12.182.66:36693)
0928 02:59:45.430487 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.430491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.430823 (+ 332us) server_negotiation.cc:407] Connection header received
0928 02:59:45.431084 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.431090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.431171 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.431234 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:45.431887 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.432710 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.434555 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.435468 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.435584 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.435634 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.435972 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.436099 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:59:45.436420 23841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.430099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:36693 (local address 127.12.182.65:54181)
0928 02:59:45.430716 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:45.430735 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:45.430883 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:45.431401 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:45.431404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:45.431475 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:45.431770 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.431779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.432839 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.432844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.434412 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.434424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.435651 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.435658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.435815 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.435850 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:59:45.435944 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:59:45.436057 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"spinlock_wait_cycles":17792,"thread_start_us":278,"threads_started":1}
I20260928 02:59:45.436669 23831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.430961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:35145 (local address 127.12.182.67:38467)
0928 02:59:45.431147 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.431153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.431406 (+ 253us) server_negotiation.cc:407] Connection header received
0928 02:59:45.431571 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.431575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.431632 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.431704 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:45.432306 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.433097 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.434813 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.435730 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.435886 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.435952 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.436320 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.436445 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:59:45.436735 23838 raft_consensus.cc:482] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:59:45.437233 23842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.430794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:38467 (local address 127.12.182.65:35145)
0928 02:59:45.431305 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:45.431319 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:45.431444 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:45.431735 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:45.431743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:45.431768 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:45.432154 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.432166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.433223 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.433227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.434676 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.434689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.435940 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.435946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.436139 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.436174 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:59:45.436818 (+ 644us) client_negotiation.cc:240] Negotiation successful
0928 02:59:45.436948 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":258,"threads_started":1}
I20260928 02:59:45.437028 23838 raft_consensus.cc:504] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } }
I20260928 02:59:45.437803 23636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0126d76a517447ec8f4d0d52279fbe64" candidate_uuid: "59bb16d1c39c443fa6118e0173b933ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" is_pre_election: true
I20260928 02:59:45.438279 23636 raft_consensus.cc:2454] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59bb16d1c39c443fa6118e0173b933ed in term 0.
I20260928 02:59:45.438357 23769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0126d76a517447ec8f4d0d52279fbe64" candidate_uuid: "59bb16d1c39c443fa6118e0173b933ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" is_pre_election: true
I20260928 02:59:45.438832 23769 raft_consensus.cc:2454] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59bb16d1c39c443fa6118e0173b933ed in term 0.
I20260928 02:59:45.438954 23838 leader_election.cc:288] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59bb16d1c39c443fa6118e0173b933ed (127.12.182.65:42583), d0b4dacd1fc84296a001c5992a5b1bb9 (127.12.182.67:38467)
I20260928 02:59:45.439014 23435 leader_election.cc:302] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59bb16d1c39c443fa6118e0173b933ed, 8281691b0f274f3cbbd3d129a4e08fdc; no voters:
I20260928 02:59:45.439543 23840 raft_consensus.cc:2781] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:59:45.439772 23840 raft_consensus.cc:482] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:59:45.439947 23840 raft_consensus.cc:3037] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:45.443064 23840 raft_consensus.cc:504] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } }
I20260928 02:59:45.443648 23840 leader_election.cc:288] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [CANDIDATE]: Term 1 election: Requested vote from peers 8281691b0f274f3cbbd3d129a4e08fdc (127.12.182.66:36693), d0b4dacd1fc84296a001c5992a5b1bb9 (127.12.182.67:38467)
I20260928 02:59:45.444144 23769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0126d76a517447ec8f4d0d52279fbe64" candidate_uuid: "59bb16d1c39c443fa6118e0173b933ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9"
I20260928 02:59:45.444330 23769 raft_consensus.cc:3037] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:45.444331 23636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0126d76a517447ec8f4d0d52279fbe64" candidate_uuid: "59bb16d1c39c443fa6118e0173b933ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8281691b0f274f3cbbd3d129a4e08fdc"
I20260928 02:59:45.444512 23636 raft_consensus.cc:3037] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:59:45.446583 23828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.439754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:41349 (local address 127.12.182.65:42583)
0928 02:59:45.439939 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.439945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.440208 (+ 263us) server_negotiation.cc:407] Connection header received
0928 02:59:45.440549 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.440554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.440623 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.440712 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:45.441200 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.442132 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.444343 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.445450 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.445575 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.445663 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.446243 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.446365 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:59:45.447232 23843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.439261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:42583 (local address 127.12.182.66:41349)
0928 02:59:45.440118 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:45.440138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:45.440311 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:45.440729 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:45.440734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:45.440762 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:45.441053 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.441064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.442242 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.442247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.444168 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.444182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.445861 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.445867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.446060 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.446097 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:59:45.446771 (+ 674us) client_negotiation.cc:240] Negotiation successful
0928 02:59:45.446893 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":569,"threads_started":1}
I20260928 02:59:45.447647 23769 raft_consensus.cc:2454] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59bb16d1c39c443fa6118e0173b933ed in term 1.
I20260928 02:59:45.447698 23636 raft_consensus.cc:2454] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59bb16d1c39c443fa6118e0173b933ed in term 1.
I20260928 02:59:45.447762 23831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.439752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:35919 (local address 127.12.182.67:38467)
0928 02:59:45.440065 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.440071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.440588 (+ 517us) server_negotiation.cc:407] Connection header received
0928 02:59:45.440776 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.440781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.440844 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.440918 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:59:45.441593 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.442619 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.444965 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.446347 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.446471 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.446535 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.447386 (+ 851us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.447510 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260928 02:59:45.447994 23502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0126d76a517447ec8f4d0d52279fbe64" candidate_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59bb16d1c39c443fa6118e0173b933ed" is_pre_election: true
I20260928 02:59:45.448333 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.439353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:38467 (local address 127.12.182.66:35919)
0928 02:59:45.440499 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0928 02:59:45.440523 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:59:45.440636 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:59:45.441060 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:59:45.441063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:59:45.441082 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:59:45.441458 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.441469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.442759 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.442764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.444791 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:59:45.444805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.447026 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.447032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:59:45.447195 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.447231 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:59:45.447906 (+ 675us) client_negotiation.cc:240] Negotiation successful
0928 02:59:45.448045 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":884,"threads_started":1}
I20260928 02:59:45.448297 23435 leader_election.cc:302] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59bb16d1c39c443fa6118e0173b933ed, 8281691b0f274f3cbbd3d129a4e08fdc; no voters:
I20260928 02:59:45.448429 23502 raft_consensus.cc:2379] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8281691b0f274f3cbbd3d129a4e08fdc in current term 1: Already voted for candidate 59bb16d1c39c443fa6118e0173b933ed in this term.
I20260928 02:59:45.448724 23840 raft_consensus.cc:2781] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:59:45.449331 23768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0126d76a517447ec8f4d0d52279fbe64" candidate_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" is_pre_election: true
I20260928 02:59:45.449707 23768 raft_consensus.cc:2379] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8281691b0f274f3cbbd3d129a4e08fdc in current term 1: Already voted for candidate 59bb16d1c39c443fa6118e0173b933ed in this term.
I20260928 02:59:45.449687 23840 raft_consensus.cc:686] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 1 LEADER]: Becoming Leader. State: Replica: 59bb16d1c39c443fa6118e0173b933ed, State: Running, Role: LEADER
I20260928 02:59:45.450155 23840 consensus_queue.cc:237] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } }
I20260928 02:59:45.450273 23569 leader_election.cc:302] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8281691b0f274f3cbbd3d129a4e08fdc; no voters: 59bb16d1c39c443fa6118e0173b933ed, d0b4dacd1fc84296a001c5992a5b1bb9
I20260928 02:59:45.450660 23838 raft_consensus.cc:2726] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:59:45.454309 23357 catalog_manager.cc:5495] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 59bb16d1c39c443fa6118e0173b933ed (127.12.182.65). New cstate: current_term: 1 leader_uuid: "59bb16d1c39c443fa6118e0173b933ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb16d1c39c443fa6118e0173b933ed" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 42583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20260928 02:59:45.538944 23828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.533069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.12.182.65:42583)
0928 02:59:45.533222 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.533226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.533448 (+ 222us) server_negotiation.cc:407] Connection header received
0928 02:59:45.533644 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.533648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.533729 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.533842 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:59:45.534239 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.534954 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.536819 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.537222 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.537326 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.537871 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:59:45.538325 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:59:45.538439 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.538574 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.538658 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:59:45.546468 23636 raft_consensus.cc:1261] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 1 FOLLOWER]: Refusing update from remote peer 59bb16d1c39c443fa6118e0173b933ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:45.546494 23768 raft_consensus.cc:1261] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [term 1 FOLLOWER]: Refusing update from remote peer 59bb16d1c39c443fa6118e0173b933ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:59:45.547233 23845 consensus_queue.cc:1035] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8281691b0f274f3cbbd3d129a4e08fdc" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:45.547537 23840 consensus_queue.cc:1035] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0b4dacd1fc84296a001c5992a5b1bb9" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:59:45.557237 23848 mvcc.cc:205] Tried to move back new op lower bound from 7334151723189313536 to 7334151722807988224. Current Snapshot: MvccSnapshot[applied={T|T < 7334151723189313536}]
I20260928 02:59:45.570948 23357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:45.571277 23357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:59:45.575439 23357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bb391031daa46b4a3edff866483fddd: Sending DeleteTablet for 3 replicas of tablet 0126d76a517447ec8f4d0d52279fbe64
I20260928 02:59:45.577111 23616 tablet_service.cc:1514] Processing DeleteTablet for tablet 0126d76a517447ec8f4d0d52279fbe64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:45 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:51860
I20260928 02:59:45.577142 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 0126d76a517447ec8f4d0d52279fbe64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:45 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:37168
I20260928 02:59:45.577783 23482 tablet_service.cc:1514] Processing DeleteTablet for tablet 0126d76a517447ec8f4d0d52279fbe64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:59:45 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:59638
I20260928 02:59:45.578500 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 23419
I20260928 02:59:45.590520 23859 tablet_replica.cc:330] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: stopping tablet replica
I20260928 02:59:45.591140 23857 tablet_replica.cc:330] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: stopping tablet replica
I20260928 02:59:45.591332 23858 tablet_replica.cc:330] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: stopping tablet replica
I20260928 02:59:45.591769 23858 raft_consensus.cc:2229] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:45.591778 23859 raft_consensus.cc:2229] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:59:45.592202 23857 raft_consensus.cc:2229] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:59:45.592334 23858 raft_consensus.cc:2258] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:45.592991 23859 raft_consensus.cc:2258] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:45.593367 23857 raft_consensus.cc:2258] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:59:45.595021 23858 ts_tablet_manager.cc:1905] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:45.595036 23859 ts_tablet_manager.cc:1905] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:45.596997 23857 ts_tablet_manager.cc:1905] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:59:45.601910 23858 ts_tablet_manager.cc:1918] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:45.601961 23859 ts_tablet_manager.cc:1918] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:45.602119 23858 log.cc:1196] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/0126d76a517447ec8f4d0d52279fbe64
I20260928 02:59:45.602252 23859 log.cc:1196] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/0126d76a517447ec8f4d0d52279fbe64
I20260928 02:59:45.602649 23858 ts_tablet_manager.cc:1939] T 0126d76a517447ec8f4d0d52279fbe64 P d0b4dacd1fc84296a001c5992a5b1bb9: Deleting consensus metadata
I20260928 02:59:45.602861 23859 ts_tablet_manager.cc:1939] T 0126d76a517447ec8f4d0d52279fbe64 P 59bb16d1c39c443fa6118e0173b933ed: Deleting consensus metadata
I20260928 02:59:45.604550 23342 catalog_manager.cc:4841] TS d0b4dacd1fc84296a001c5992a5b1bb9 (127.12.182.67:38467): tablet 0126d76a517447ec8f4d0d52279fbe64 (table test-table [id=c70e70f66644439ca9ac6e102a61bd32]) successfully deleted
I20260928 02:59:45.604816 23345 catalog_manager.cc:4841] TS 59bb16d1c39c443fa6118e0173b933ed (127.12.182.65:42583): tablet 0126d76a517447ec8f4d0d52279fbe64 (table test-table [id=c70e70f66644439ca9ac6e102a61bd32]) successfully deleted
Sep 28 02:59:45 dist-test-slave-k5rg krb5kdc[23313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 02:59:45.606875 23857 ts_tablet_manager.cc:1918] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:59:45.607038 23857 log.cc:1196] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/0126d76a517447ec8f4d0d52279fbe64
I20260928 02:59:45.607519 23857 ts_tablet_manager.cc:1939] T 0126d76a517447ec8f4d0d52279fbe64 P 8281691b0f274f3cbbd3d129a4e08fdc: Deleting consensus metadata
I20260928 02:59:45.609128 23342 catalog_manager.cc:4841] TS 8281691b0f274f3cbbd3d129a4e08fdc (127.12.182.66:36693): tablet 0126d76a517447ec8f4d0d52279fbe64 (table test-table [id=c70e70f66644439ca9ac6e102a61bd32]) successfully deleted
I20260928 02:59:45.613545 23828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.580063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.12.182.65:42583)
0928 02:59:45.580252 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.580258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.589668 (+ 9410us) server_negotiation.cc:407] Connection header received
0928 02:59:45.589867 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.589874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.589970 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.590093 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:45.592181 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.593326 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.601443 (+ 8117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.602176 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.602292 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.607693 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:45.607707 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:45.607724 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:45.607775 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:45.610954 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:45.611727 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:45.611735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:45.611737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:45.611853 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:45.612253 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:45.612256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:45.612258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:45.612607 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:45.612846 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.613153 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.613324 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:59:45.614303 23542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:45.709667 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 23419
I20260928 02:59:45.724583 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 23553
Sep 28 02:59:45 dist-test-slave-k5rg krb5kdc[23313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 02:59:45.738338 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.725258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.12.182.66:36693)
0928 02:59:45.725450 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.725456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.725511 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:59:45.725728 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.725734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.725800 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.725895 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:45.727123 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.727799 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.728974 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.729361 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.729476 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.733173 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:45.733185 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:45.733200 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:45.733231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:45.735766 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:45.736383 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:45.736390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:45.736392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:45.736524 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:45.736858 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:45.736862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:45.736864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:45.737215 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:45.737488 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.737795 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.738004 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:59:45.738950 23676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:45.826323 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 23553
I20260928 02:59:45.840283 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 23686
Sep 28 02:59:45 dist-test-slave-k5rg krb5kdc[23313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 02:59:45.853749 23831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:59:45.840932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.12.182.67:38467)
0928 02:59:45.841089 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:59:45.841093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:59:45.841112 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:59:45.841295 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:59:45.841299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:59:45.841381 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:59:45.841461 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:59:45.842673 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.843324 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.844412 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:59:45.844812 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:59:45.844896 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:59:45.848850 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:59:45.848865 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:59:45.848880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:59:45.848914 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:59:45.851501 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:45.852108 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:45.852114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:45.852116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:45.852238 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:59:45.852582 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:59:45.852584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:59:45.852586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:59:45.852894 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:59:45.853115 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 02:59:45.853324 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 02:59:45.853475 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:59:45.854323 23809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:45.939357 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 23686
I20260928 02:59:45.952236 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 23326
I20260928 02:59:45.953265 23388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:59:46.057402 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 23326
2026-09-28T02:59:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4229 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:59:46 dist-test-slave-k5rg krb5kdc[23867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:59:46 dist-test-slave-k5rg krb5kdc[23867](info): set up 2 sockets
Sep 28 02:59:46 dist-test-slave-k5rg krb5kdc[23867](info): commencing operation
krb5kdc: starting...
W20260928 02:59:47.066386 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 02:59:47 dist-test-slave-k5rg krb5kdc[23867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:59:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:59:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/rangeradmin_127.12.182.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/rangeradmin_127.12.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/rangerlookup_127.12.182.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/rangerlookup_127.12.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/HTTP_127.12.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/HTTP_127.12.182.84@KRBTEST.COM.keytab.
I20260928 02:59:47.351831 13017 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmphc7w/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/postgres -l logfile start
2026-09-28 02:59:49.695 UTC [23897] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:59:49.695 UTC [23897] LOG: listening on IPv4 address "127.12.182.84", port 59617
2026-09-28 02:59:49.699 UTC [23897] LOG: listening on Unix socket "/tmp/.s.PGSQL.59617"
2026-09-28 02:59:49.718 UTC [23900] LOG: database system was shut down at 2026-09-28 02:59:48 UTC
2026-09-28 02:59:49.721 UTC [23897] LOG: database system is ready to accept connections
I20260928 02:59:50.610013 13017 mini_postgres.cc:96] Postgres bound to 59617
2026-09-28 02:59:50.626 UTC [23908] FATAL: database "slave" does not exist
127.12.182.84:59617 - accepting connections
I20260928 02:59:50.627035 13017 mini_ranger.cc:161] Starting Ranger...
I20260928 02:59:50.649905 13017 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:59:51.094280 13017 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:59:51.094410 13017 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-admin
2026-09-28 02:59:51,391 [I] DB FLAVOR :POSTGRES
2026-09-28 02:59:51,392 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:59:51,392 [I] Checking connection..
2026-09-28 02:59:51,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:59:51,643 [I] Checking connection passed.
2026-09-28 02:59:51,643 [I] --------- Verifying version history table ---------
2026-09-28 02:59:51,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:59:51,870 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:59:51,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:59:52,120 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:59:52,120 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:59:52,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:59:52.364 UTC [23972] WARNING: there is no transaction in progress
2026-09-28 02:59:52,369 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:59:52,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:59:52,603 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:59:52,603 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:59:52,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:59:52,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:59:53,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k5rg.c.gcp-upstream.internal','N') ;"
2026-09-28 02:59:53,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:59:53,582 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:59:53,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:59:53,813 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:59:53,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:59:54,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:59:54,270 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:59:54,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:59:55.188 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.203 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.218 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.232 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.246 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.263 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.315 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.321 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.334 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.343 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.353 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.364 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.373 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.379 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.385 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55.391 UTC [24080] WARNING: there is no transaction in progress
2026-09-28 02:59:55,985 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:59:55,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:59:56,225 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:59:56,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:59:56,480 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:59:56,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:59:56,732 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:59:56,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:59:56,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:59:57,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k5rg.c.gcp-upstream.internal';"
2026-09-28 02:59:57,402 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:59:57,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:59:57,655 [I] DB_PATCHES have already been applied
I20260928 02:59:57.661185 13017 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmphc7w/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:59:57.661270 13017 mini_ranger.cc:193] Using host: 127.12.182.84
I20260928 02:59:57.674325 13017 mini_ranger.cc:227] Ranger admin URL: http://127.12.182.84:37633
Sep 28, 2026 2:59:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:59:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:59:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:59:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:59:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:59:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:59:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:59:58.686741 13017 mini_ranger.cc:160] Time spent starting Ranger: real 8.060s user 0.005s sys 0.031s
Sep 28, 2026 3:00:18 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:00:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 3:00:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:00:18 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:00:19 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 03:00:19.901199 13017 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/rangerkms_127.12.182.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/rangerkms_127.12.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.182.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.182.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.182.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/HTTP_127.12.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.182.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/HTTP_127.12.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 03:00:20 dist-test-slave-k5rg krb5kdc[23867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564420, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 03:00:20.075719 13017 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 03:00:20.098261 13017 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 03:00:20.934792 13017 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 03:00:20.934939 13017 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-kms
I20260928 03:00:20.935075 13017 mini_ranger_kms.cc:227] Using postgres at 127.12.182.84:59617
2026-09-28 03:00:21,334 [I] DB FLAVOR :POSTGRES
2026-09-28 03:00:21,334 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 03:00:21,334 [I] Checking connection
2026-09-28 03:00:21,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 03:00:21,564 [I] connection success
2026-09-28 03:00:21,565 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 03:00:21,565 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 03:00:21,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 03:00:21,781 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 03:00:21,782 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 03:00:21,782 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 03:00:21,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:59617/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 03:00:22,021 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 03:00:22.445685 13017 mini_ranger_kms.cc:310] Created kms service
I20260928 03:00:22.637405 13017 mini_ranger_kms.cc:326] Created kudu user
I20260928 03:00:22.679963 13017 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 03:00:22 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564420, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:00:23.068783 13017 mini_ranger_kms.cc:384] Added ranger policy
I20260928 03:00:23.069119 13017 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 03:00:23.069172 13017 mini_ranger_kms.cc:240] Using host: 127.12.182.84
I20260928 03:00:23.085320 13017 mini_ranger_kms.cc:276] Ranger KMS PID: 24260
I20260928 03:00:23.085510 13017 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.182.84:42609
Sep 28, 2026 3:00:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:00:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:00:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:00:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:00:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:00:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 03:00:25.174278 13017 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.099s user 0.135s sys 0.280s
I20260928 03:00:25.174522 13017 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 03:00:25.174729 13017 mini_ranger_kms.cc:401] 127.12.182.84:42609/kms/v1/keys
Sep 28 03:00:25 dist-test-slave-k5rg krb5kdc[23867](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.182.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 03:00:25 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.182.84@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
Sep 28 03:00:25 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.182.84@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
Sep 28 03:00:28 dist-test-slave-k5rg krb5kdc[23867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:00:28.101631 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:40713
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:40713
--ranger_config_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:42609/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:00:28.408474 24311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:00:28.408769 24311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:00:28.408911 24311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:00:28.416002 24311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:00:28.416118 24311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:00:28.416199 24311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:00:28.416275 24311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:00:28.416339 24311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:00:28.426213 24311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.182.84:42609/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:40713
--ranger_config_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:40713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:00:28.427424 24311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:00:28.428700 24311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:00:28.439927 24316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:00:28.439955 24319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:00:28.439975 24317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:00:28.440443 24311 server_base.cc:1034] running on GCE node
I20260928 03:00:28.441190 24311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:00:28.442484 24311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:00:28.443683 24311 hybrid_clock.cc:648] HybridClock initialized: now 1790564428443645 us; error 31 us; skew 500 ppm
I20260928 03:00:28.444057 24311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:00:28 dist-test-slave-k5rg krb5kdc[23867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564428, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:00:28.450428 24311 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:00:28.451670 24311 webserver.cc:460] Webserver started at http://127.12.182.126:36831/ using document root <none> and password file <none>
I20260928 03:00:28.452221 24311 fs_manager.cc:359] Metadata directory not provided
I20260928 03:00:28.452307 24311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:00:28.452530 24311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:00:28 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564428, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:00:28.501475 24311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "e148f0c3842f4d81b207458728686b1a"
format_stamp: "Formatted at 2026-09-28 03:00:28 on dist-test-slave-k5rg"
server_key: "45f269db986ada7d7bc78fb21658ef9e"
server_key_iv: "4590670efa26a80ba718753e76e53321"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:28.502429 24311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "e148f0c3842f4d81b207458728686b1a"
format_stamp: "Formatted at 2026-09-28 03:00:28 on dist-test-slave-k5rg"
server_key: "45f269db986ada7d7bc78fb21658ef9e"
server_key_iv: "4590670efa26a80ba718753e76e53321"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:28.507735 24311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 03:00:28.523018 24326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:00:28.524433 24311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:00:28.524600 24311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "e148f0c3842f4d81b207458728686b1a"
format_stamp: "Formatted at 2026-09-28 03:00:28 on dist-test-slave-k5rg"
server_key: "45f269db986ada7d7bc78fb21658ef9e"
server_key_iv: "4590670efa26a80ba718753e76e53321"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:28.524806 24311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:00:28.549907 24311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:00:28.553753 24311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:00:28.554019 24311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:00:28.576967 24311 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:40713
I20260928 03:00:28.576977 24377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:40713 every 8 connection(s)
I20260928 03:00:28.578281 24311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:00:28.581916 24378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:00:28.586366 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 24311
I20260928 03:00:28.586555 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:00:28.587174 13017 external_mini_cluster.cc:1428] Setting key 6fd843f1b240f05751eda5983c72c5b4
I20260928 03:00:28.591873 24378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a: Bootstrap starting.
I20260928 03:00:28.595194 24378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a: Neither blocks nor log segments found. Creating new log.
I20260928 03:00:28.596314 24378 log.cc:824] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a: Log is configured to *not* fsync() on all Append() calls
I20260928 03:00:28.598838 24378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a: No bootstrap required, opened a new log
Sep 28 03:00:28 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:00:28.606027 24378 raft_consensus.cc:348] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e148f0c3842f4d81b207458728686b1a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40713 } }
I20260928 03:00:28.606393 24378 raft_consensus.cc:374] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:00:28.606492 24378 raft_consensus.cc:729] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148f0c3842f4d81b207458728686b1a, State: Initialized, Role: FOLLOWER
I20260928 03:00:28.607051 24378 consensus_queue.cc:260] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e148f0c3842f4d81b207458728686b1a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40713 } }
I20260928 03:00:28.607263 24378 raft_consensus.cc:388] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:00:28.607385 24378 raft_consensus.cc:482] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:00:28.607542 24378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:00:28.607846 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:28.589071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51806 (local address 127.12.182.126:40713)
0928 03:00:28.589873 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:28.589884 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:28.589926 (+ 42us) server_negotiation.cc:407] Connection header received
0928 03:00:28.591464 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:28.591486 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:28.591843 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:28.592153 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:00:28.593687 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:28.594975 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:28.596612 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:28.597284 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:28.597472 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:28.601131 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:00:28.601141 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:00:28.601158 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:00:28.601193 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:00:28.604023 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:28.604651 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:28.604657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:28.604666 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:28.604807 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:28.605155 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:28.605159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:28.605161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:28.605694 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:00:28.605958 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:28.606424 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 03:00:28.606815 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":365,"threads_started":1}
I20260928 03:00:28.609730 24378 raft_consensus.cc:504] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e148f0c3842f4d81b207458728686b1a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40713 } }
I20260928 03:00:28.610183 24378 leader_election.cc:302] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e148f0c3842f4d81b207458728686b1a; no voters:
I20260928 03:00:28.610811 24378 leader_election.cc:288] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:00:28.610905 24383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:00:28.611495 24383 raft_consensus.cc:686] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [term 1 LEADER]: Becoming Leader. State: Replica: e148f0c3842f4d81b207458728686b1a, State: Running, Role: LEADER
I20260928 03:00:28.612106 24383 consensus_queue.cc:237] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e148f0c3842f4d81b207458728686b1a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40713 } }
I20260928 03:00:28.612818 24378 sys_catalog.cc:564] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:00:28.614770 24385 sys_catalog.cc:455] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e148f0c3842f4d81b207458728686b1a. Latest consensus state: current_term: 1 leader_uuid: "e148f0c3842f4d81b207458728686b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e148f0c3842f4d81b207458728686b1a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40713 } } }
I20260928 03:00:28.615155 24385 sys_catalog.cc:458] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [sys.catalog]: This master's current role is: LEADER
I20260928 03:00:28.615794 24384 sys_catalog.cc:455] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e148f0c3842f4d81b207458728686b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e148f0c3842f4d81b207458728686b1a" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40713 } } }
I20260928 03:00:28.616173 24384 sys_catalog.cc:458] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a [sys.catalog]: This master's current role is: LEADER
I20260928 03:00:28.616511 24391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:00:28.621457 24391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:00:28.626585 24378 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 03:00:28.640311 24391 catalog_manager.cc:1347] Generated new cluster ID: db7bd006229b4651a16f8dc504bcd351
I20260928 03:00:28.640458 24391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:00:28.675545 24391 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:00:28.677414 24391 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:00:28.692757 24391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a: Generated new TSK 0
I20260928 03:00:28.693845 24391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 03:00:29.385677 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:42609/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40713
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:00:29 dist-test-slave-k5rg krb5kdc[23867](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564429, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 03:00:29.826472 24423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:00:29.826853 24423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:00:29.827062 24423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:00:29.838798 24423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:00:29.839004 24423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:00:29.839211 24423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:00:29.854871 24423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.182.84:42609/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:00:29.856766 24423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:00:29.858620 24423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:00:29.875782 24433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:00:29.876386 24434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:00:29.880977 24436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:00:29.880980 24423 server_base.cc:1034] running on GCE node
I20260928 03:00:29.881824 24423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:00:29.882897 24423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:00:29.884155 24423 hybrid_clock.cc:648] HybridClock initialized: now 1790564429884086 us; error 61 us; skew 500 ppm
I20260928 03:00:29.884513 24423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:00:29 dist-test-slave-k5rg krb5kdc[23867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564429, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:00:29.888422 24423 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:00:29.906839 24423 webserver.cc:460] Webserver started at http://127.12.182.65:32875/ using document root <none> and password file <none>
I20260928 03:00:29.907603 24423 fs_manager.cc:359] Metadata directory not provided
I20260928 03:00:29.907732 24423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:00:29.908118 24423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:00:29 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564429, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:00:29.933283 24423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "a6c1359266ff4e558ed1263b72b3dd27"
format_stamp: "Formatted at 2026-09-28 03:00:29 on dist-test-slave-k5rg"
server_key: "2941ac457c021d9d28a4e437fabd6da4"
server_key_iv: "9547da6ae1a0824d02087c2b3d9888ae"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:29.934196 24423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "a6c1359266ff4e558ed1263b72b3dd27"
format_stamp: "Formatted at 2026-09-28 03:00:29 on dist-test-slave-k5rg"
server_key: "2941ac457c021d9d28a4e437fabd6da4"
server_key_iv: "9547da6ae1a0824d02087c2b3d9888ae"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:29.940034 24423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:00:29.966073 24443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:00:29.968071 24423 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.003s
I20260928 03:00:29.968358 24423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "a6c1359266ff4e558ed1263b72b3dd27"
format_stamp: "Formatted at 2026-09-28 03:00:29 on dist-test-slave-k5rg"
server_key: "2941ac457c021d9d28a4e437fabd6da4"
server_key_iv: "9547da6ae1a0824d02087c2b3d9888ae"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:29.968708 24423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:00:30.018375 24423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:00:30.023737 24423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:00:30.024171 24423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:00:30.031718 24423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:00:30.039852 24423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:00:30.040011 24423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:00:30.040221 24423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:00:30.040367 24423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:00:30.102880 24423 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:39715
I20260928 03:00:30.102898 24555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:39715 every 8 connection(s)
I20260928 03:00:30.104334 24423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:00:30.106444 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 24423
I20260928 03:00:30.106657 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 03:00:30.107457 13017 external_mini_cluster.cc:1428] Setting key 036b866f562837b7028ece1dd097478e
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
Sep 28 03:00:30 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564429, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
Sep 28 03:00:30 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564429, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 03:00:30.206070 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:42609/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40713
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:00:30.218426 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:30.140372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:47813 (local address 127.12.182.126:40713)
0928 03:00:30.171100 (+ 30728us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:30.171108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:30.173759 (+ 2651us) server_negotiation.cc:407] Connection header received
0928 03:00:30.178750 (+ 4991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:30.178758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:30.178886 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:30.178965 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:00:30.181122 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:30.182309 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:30.186023 (+ 3714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:30.186817 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:30.186911 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:30.201012 (+ 14101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:00:30.201020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:00:30.201024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:00:30.201079 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:00:30.204462 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:30.215364 (+ 10902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:30.215371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:30.215375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:30.215544 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:30.216163 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:30.216168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:30.216172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:30.216581 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:00:30.216720 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:30.217789 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0928 03:00:30.218043 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30578,"thread_start_us":33239,"threads_started":1}
I20260928 03:00:30.221004 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:30.140943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40713 (local address 127.12.182.65:47813)
0928 03:00:30.171734 (+ 30791us) negotiation.cc:107] Waiting for socket to connect
0928 03:00:30.171805 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 03:00:30.178349 (+ 6544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:00:30.179609 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:00:30.179622 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:00:30.180244 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:00:30.180933 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:30.180954 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:30.183267 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:30.183272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:30.185822 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:30.185841 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:30.187215 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:30.187223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:30.187574 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:30.197370 (+ 9796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:00:30.197413 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:00:30.200472 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:00:30.214438 (+ 13966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:00:30.214450 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:00:30.214473 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:00:30.215180 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:00:30.215726 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:00:30.215731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:00:30.215734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:00:30.216015 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:00:30.216816 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:00:30.216824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:00:30.217280 (+ 456us) client_negotiation.cc:769] Sending connection context
0928 03:00:30.218964 (+ 1684us) client_negotiation.cc:240] Negotiation successful
0928 03:00:30.219541 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30451,"thread_start_us":30227,"threads_started":1}
I20260928 03:00:30.223255 24556 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40713
I20260928 03:00:30.223584 24556 heartbeater.cc:464] Registering TS with master...
I20260928 03:00:30.224408 24556 heartbeater.cc:511] Master 127.12.182.126:40713 requested a full tablet report, sending...
I20260928 03:00:30.226907 24343 ts_manager.cc:194] Registered new tserver with Master: a6c1359266ff4e558ed1263b72b3dd27 (127.12.182.65:39715)
I20260928 03:00:30.228916 24343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:47813
Sep 28 03:00:30 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564429, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
W20260928 03:00:30.676337 24562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:00:30.676779 24562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:00:30.676990 24562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:00:30.687359 24562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:00:30.687480 24562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:00:30.687695 24562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:00:30.700984 24562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.182.84:42609/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:00:30.702670 24562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:00:30.703912 24562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:00:30.718801 24569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:00:30.718873 24570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:00:30.730919 24572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:00:30.731019 24562 server_base.cc:1034] running on GCE node
I20260928 03:00:30.731667 24562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:00:30.733772 24562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:00:30.735071 24562 hybrid_clock.cc:648] HybridClock initialized: now 1790564430734960 us; error 94 us; skew 500 ppm
I20260928 03:00:30.735466 24562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:00:30 dist-test-slave-k5rg krb5kdc[23867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564430, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:00:30.740473 24562 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:00:30.752434 24562 webserver.cc:460] Webserver started at http://127.12.182.66:44353/ using document root <none> and password file <none>
I20260928 03:00:30.753394 24562 fs_manager.cc:359] Metadata directory not provided
I20260928 03:00:30.753542 24562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:00:30.753968 24562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:00:30 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564430, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:00:30.770639 24562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "66bd28798e334ff18e9bf732ae428fa1"
format_stamp: "Formatted at 2026-09-28 03:00:30 on dist-test-slave-k5rg"
server_key: "1d6ebe04996edf42153cb4eef6227cab"
server_key_iv: "aa21bb16db228071f5ecbd03196d9ce7"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:30.771703 24562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "66bd28798e334ff18e9bf732ae428fa1"
format_stamp: "Formatted at 2026-09-28 03:00:30 on dist-test-slave-k5rg"
server_key: "1d6ebe04996edf42153cb4eef6227cab"
server_key_iv: "aa21bb16db228071f5ecbd03196d9ce7"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:30.777942 24562 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 03:00:30.812202 24586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:00:30.814429 24562 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.003s sys 0.009s
I20260928 03:00:30.814620 24562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "66bd28798e334ff18e9bf732ae428fa1"
format_stamp: "Formatted at 2026-09-28 03:00:30 on dist-test-slave-k5rg"
server_key: "1d6ebe04996edf42153cb4eef6227cab"
server_key_iv: "aa21bb16db228071f5ecbd03196d9ce7"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:30.814918 24562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:00:30.874946 24562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:00:30.879928 24562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:00:30.880247 24562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:00:30.881405 24562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:00:30.883077 24562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:00:30.883168 24562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:00:30.883337 24562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:00:30.883402 24562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:00:30.920799 24562 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:43723
I20260928 03:00:30.920876 24698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:43723 every 8 connection(s)
I20260928 03:00:30.922416 24562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:00:30.926765 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 24562
I20260928 03:00:30.926918 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 03:00:30.927407 13017 external_mini_cluster.cc:1428] Setting key 3744942eb344f5683f169ec4dc085681
Sep 28 03:00:30 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564430, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:00:30.954348 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:30.926169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:60145 (local address 127.12.182.126:40713)
0928 03:00:30.934041 (+ 7872us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:30.934048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:30.937192 (+ 3144us) server_negotiation.cc:407] Connection header received
0928 03:00:30.937251 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:30.937256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:30.937336 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:30.937425 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:00:30.939306 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:30.940405 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:30.942684 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:30.943301 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:30.943390 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:30.947610 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:00:30.947617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:00:30.947622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:00:30.947658 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:00:30.950756 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:30.951583 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:30.951588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:30.951591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:30.951719 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:30.952186 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:30.952191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:30.952194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:30.952597 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:00:30.952729 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:30.953746 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0928 03:00:30.953980 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7740,"thread_start_us":7637,"threads_started":1}
I20260928 03:00:30.956661 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:30.925669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40713 (local address 127.12.182.66:60145)
0928 03:00:30.934505 (+ 8836us) negotiation.cc:107] Waiting for socket to connect
0928 03:00:30.934556 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 03:00:30.936801 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:00:30.937818 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:00:30.937829 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:00:30.938398 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:00:30.939127 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:30.939150 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:30.940590 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:30.940595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:30.942535 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:30.942548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:30.943706 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:30.943711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:30.944041 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:30.944751 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:00:30.944779 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:00:30.947376 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:00:30.950906 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:00:30.950914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:00:30.950931 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:00:30.951458 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:00:30.951841 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:00:30.951845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:00:30.951848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:00:30.952071 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:00:30.952823 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:00:30.952831 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:00:30.953280 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 03:00:30.954864 (+ 1584us) client_negotiation.cc:240] Negotiation successful
0928 03:00:30.955372 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8564,"thread_start_us":17935,"threads_started":1}
I20260928 03:00:30.958509 24699 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40713
I20260928 03:00:30.958806 24699 heartbeater.cc:464] Registering TS with master...
I20260928 03:00:30.959571 24699 heartbeater.cc:511] Master 127.12.182.126:40713 requested a full tablet report, sending...
I20260928 03:00:30.961179 24343 ts_manager.cc:194] Registered new tserver with Master: 66bd28798e334ff18e9bf732ae428fa1 (127.12.182.66:43723)
I20260928 03:00:30.962201 24343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:60145
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 03:00:30.998299 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:42609/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40713
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:00:31.232604 24556 heartbeater.cc:503] Master 127.12.182.126:40713 was elected leader, sending a full tablet report...
W20260928 03:00:31.358762 24705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:00:31.359046 24705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:00:31.359166 24705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:00:31.366359 24705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:00:31.366477 24705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:00:31.366619 24705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:00:31.376371 24705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.182.84:42609/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:00:31.377581 24705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:00:31.378794 24705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:00:31.390800 24713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:00:31.390874 24710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:00:31.390853 24711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:00:31.391820 24705 server_base.cc:1034] running on GCE node
I20260928 03:00:31.392297 24705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:00:31.393121 24705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:00:31.394315 24705 hybrid_clock.cc:648] HybridClock initialized: now 1790564431394232 us; error 70 us; skew 500 ppm
I20260928 03:00:31.394600 24705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:00:31 dist-test-slave-k5rg krb5kdc[23867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564431, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:00:31.397524 24705 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:00:31.398867 24705 webserver.cc:460] Webserver started at http://127.12.182.67:33087/ using document root <none> and password file <none>
I20260928 03:00:31.399488 24705 fs_manager.cc:359] Metadata directory not provided
I20260928 03:00:31.399603 24705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:00:31.399881 24705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:00:31 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564431, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:00:31.413700 24705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "e006f07a51a34f9f8fabfe745853dcf3"
format_stamp: "Formatted at 2026-09-28 03:00:31 on dist-test-slave-k5rg"
server_key: "e9bdf2faaa9d526bf80122030d3d16fc"
server_key_iv: "e346bb0a1cdf05a5b343a1fa6265e3ce"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:31.414503 24705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "e006f07a51a34f9f8fabfe745853dcf3"
format_stamp: "Formatted at 2026-09-28 03:00:31 on dist-test-slave-k5rg"
server_key: "e9bdf2faaa9d526bf80122030d3d16fc"
server_key_iv: "e346bb0a1cdf05a5b343a1fa6265e3ce"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:31.419447 24705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260928 03:00:31.434762 24720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:00:31.436281 24705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:00:31.436440 24705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "e006f07a51a34f9f8fabfe745853dcf3"
format_stamp: "Formatted at 2026-09-28 03:00:31 on dist-test-slave-k5rg"
server_key: "e9bdf2faaa9d526bf80122030d3d16fc"
server_key_iv: "e346bb0a1cdf05a5b343a1fa6265e3ce"
server_key_version: "kuduclusterkey@0"
I20260928 03:00:31.436643 24705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:00:31.449443 24705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:00:31.452698 24705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:00:31.452960 24705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:00:31.453816 24705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:00:31.455070 24705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:00:31.455153 24705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:00:31.455265 24705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:00:31.455317 24705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:00:31.483790 24705 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:42171
I20260928 03:00:31.483835 24832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:42171 every 8 connection(s)
I20260928 03:00:31.485016 24705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:00:31.485595 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 24705
I20260928 03:00:31.485813 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 03:00:31.486541 13017 external_mini_cluster.cc:1428] Setting key c397d8d080b77841d22b082927173cd6
Sep 28 03:00:31 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564431, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:00:31.504535 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.487824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:38511 (local address 127.12.182.126:40713)
0928 03:00:31.488561 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:31.488569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:31.489650 (+ 1081us) server_negotiation.cc:407] Connection header received
0928 03:00:31.491673 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:31.491677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:31.491755 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:31.491845 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:00:31.493271 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.494051 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.495533 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.495974 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.496075 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.499153 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:00:31.499158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:00:31.499161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:00:31.499200 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:00:31.501702 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:31.502426 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:31.502431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:31.502434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:31.502593 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:31.503106 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:31.503111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:31.503114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:31.503396 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:00:31.503501 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:31.504133 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 03:00:31.504289 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":325,"threads_started":1}
I20260928 03:00:31.505383 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.488543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40713 (local address 127.12.182.67:38511)
0928 03:00:31.489481 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0928 03:00:31.489533 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 03:00:31.491434 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:00:31.492069 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:00:31.492078 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:00:31.492497 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:00:31.493095 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.493113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.494186 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.494190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.495391 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.495402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.496156 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.496160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.496354 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.496983 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:00:31.497005 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:00:31.498939 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:00:31.501811 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:00:31.501818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:00:31.501836 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:00:31.502323 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:00:31.502690 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:00:31.502694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:00:31.502698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:00:31.502989 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:00:31.503490 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:00:31.503495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:00:31.503790 (+ 295us) client_negotiation.cc:769] Sending connection context
0928 03:00:31.504043 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 03:00:31.504399 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":375,"threads_started":1}
I20260928 03:00:31.506563 24833 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40713
I20260928 03:00:31.506855 24833 heartbeater.cc:464] Registering TS with master...
I20260928 03:00:31.507514 24833 heartbeater.cc:511] Master 127.12.182.126:40713 requested a full tablet report, sending...
I20260928 03:00:31.508817 24343 ts_manager.cc:194] Registered new tserver with Master: e006f07a51a34f9f8fabfe745853dcf3 (127.12.182.67:42171)
I20260928 03:00:31.509563 24343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:38511
I20260928 03:00:31.515213 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:00:31.530805 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.519450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.12.182.126:40713)
0928 03:00:31.519604 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:31.519609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:31.519974 (+ 365us) server_negotiation.cc:407] Connection header received
0928 03:00:31.520118 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:31.520121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:31.520202 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:31.520275 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:00:31.521724 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.522424 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.523698 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.524119 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.524206 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.526442 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:00:31.526447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:00:31.526450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:00:31.526480 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:00:31.528769 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:31.529348 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:31.529351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:31.529353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:31.529470 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:31.529796 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:31.529799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:31.529801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:31.530045 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:00:31.530216 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:31.530426 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 03:00:31.530575 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 03:00:31.534476 24343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:00:31.541311 24343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:00:31.562579 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.552223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:43723 (local address 127.0.0.1:54910)
0928 03:00:31.553265 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0928 03:00:31.553300 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 03:00:31.553468 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:00:31.554379 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:00:31.554387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:00:31.554419 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:00:31.555080 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.555093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.556934 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.556940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.559388 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.559403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.561428 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.561438 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.561788 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.561848 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 03:00:31.561985 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 03:00:31.562173 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":48,"thread_start_us":1953,"threads_started":1}
I20260928 03:00:31.563294 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.552172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.12.182.66:43723)
0928 03:00:31.552713 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:31.552722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:31.553401 (+ 679us) server_negotiation.cc:407] Connection header received
0928 03:00:31.553676 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:31.553685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:31.553929 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:31.554108 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:00:31.555271 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.556775 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.559586 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.560954 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.561085 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.561275 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:31.562828 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0928 03:00:31.562993 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":294,"threads_started":1}
I20260928 03:00:31.564141 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.552223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39715 (local address 127.0.0.1:52874)
0928 03:00:31.554429 (+ 2206us) negotiation.cc:107] Waiting for socket to connect
0928 03:00:31.554444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:00:31.554543 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:00:31.555368 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:00:31.555373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:00:31.555396 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:00:31.555741 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.555753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.557530 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.557535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.559596 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.559608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.561492 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.561498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.563583 (+ 2085us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.563612 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 03:00:31.563682 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 03:00:31.563776 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2091,"mutex_wait_us":1842,"thread_start_us":706,"threads_started":1}
I20260928 03:00:31.564260 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.552507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.12.182.65:39715)
0928 03:00:31.554039 (+ 1532us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:31.554047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:31.554649 (+ 602us) server_negotiation.cc:407] Connection header received
0928 03:00:31.554744 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:31.554752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:31.554969 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:31.555141 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:00:31.555918 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.557385 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.559750 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.561075 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.561184 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.561370 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:31.563758 (+ 2388us) server_negotiation.cc:299] Negotiation successful
0928 03:00:31.563955 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":1863,"threads_started":1}
I20260928 03:00:31.567749 24634 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca19b8971874374bd694693d5369d8e (DEFAULT_TABLE table=test-table [id=c3287c278a954f7c88cec5eff08bfb44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:00:31.568106 24491 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca19b8971874374bd694693d5369d8e (DEFAULT_TABLE table=test-table [id=c3287c278a954f7c88cec5eff08bfb44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:00:31.569291 24634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca19b8971874374bd694693d5369d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:00:31.569437 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca19b8971874374bd694693d5369d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:00:31.571135 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.554560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:42171 (local address 127.0.0.1:33134)
0928 03:00:31.562114 (+ 7554us) negotiation.cc:107] Waiting for socket to connect
0928 03:00:31.562130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:00:31.562219 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:00:31.563316 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:00:31.563320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:00:31.563342 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:00:31.563745 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.563758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.566324 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.566330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.568338 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.568353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.570342 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.570349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.570524 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.570556 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 03:00:31.570641 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 03:00:31.570763 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7445,"mutex_wait_us":23,"thread_start_us":7308,"threads_started":1}
I20260928 03:00:31.571825 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.554898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.12.182.67:42171)
0928 03:00:31.562715 (+ 7817us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:31.562724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:31.562758 (+ 34us) server_negotiation.cc:407] Connection header received
0928 03:00:31.562844 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:31.562851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:31.563101 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:31.563288 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:00:31.564018 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.566157 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.568531 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.569880 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.570000 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.570212 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:31.571335 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0928 03:00:31.571509 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7655,"thread_start_us":7504,"threads_started":1}
I20260928 03:00:31.575408 24768 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca19b8971874374bd694693d5369d8e (DEFAULT_TABLE table=test-table [id=c3287c278a954f7c88cec5eff08bfb44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:00:31.577062 24768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca19b8971874374bd694693d5369d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:00:31.587520 24853 tablet_bootstrap.cc:492] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: Bootstrap starting.
I20260928 03:00:31.594401 24853 tablet_bootstrap.cc:654] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: Neither blocks nor log segments found. Creating new log.
I20260928 03:00:31.595319 24852 tablet_bootstrap.cc:492] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: Bootstrap starting.
I20260928 03:00:31.596644 24853 log.cc:824] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: Log is configured to *not* fsync() on all Append() calls
I20260928 03:00:31.598304 24854 tablet_bootstrap.cc:492] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: Bootstrap starting.
I20260928 03:00:31.599475 24852 tablet_bootstrap.cc:654] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: Neither blocks nor log segments found. Creating new log.
I20260928 03:00:31.600710 24852 log.cc:824] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: Log is configured to *not* fsync() on all Append() calls
I20260928 03:00:31.603106 24854 tablet_bootstrap.cc:654] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: Neither blocks nor log segments found. Creating new log.
I20260928 03:00:31.604208 24854 log.cc:824] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:00:31.604916 24852 tablet_bootstrap.cc:492] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: No bootstrap required, opened a new log
I20260928 03:00:31.605118 24853 tablet_bootstrap.cc:492] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: No bootstrap required, opened a new log
I20260928 03:00:31.605217 24852 ts_tablet_manager.cc:1397] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 03:00:31.605361 24853 ts_tablet_manager.cc:1397] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260928 03:00:31.606981 24854 tablet_bootstrap.cc:492] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: No bootstrap required, opened a new log
I20260928 03:00:31.607208 24854 ts_tablet_manager.cc:1397] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260928 03:00:31.612599 24853 raft_consensus.cc:348] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bd28798e334ff18e9bf732ae428fa1" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43723 } } peers { permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 } } peers { permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 } }
I20260928 03:00:31.613042 24853 raft_consensus.cc:374] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:00:31.613135 24853 raft_consensus.cc:729] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bd28798e334ff18e9bf732ae428fa1, State: Initialized, Role: FOLLOWER
I20260928 03:00:31.613722 24853 consensus_queue.cc:260] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bd28798e334ff18e9bf732ae428fa1" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43723 } } peers { permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 } } peers { permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 } }
I20260928 03:00:31.614744 24699 heartbeater.cc:503] Master 127.12.182.126:40713 was elected leader, sending a full tablet report...
I20260928 03:00:31.615351 24853 ts_tablet_manager.cc:1428] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260928 03:00:31.615141 24852 raft_consensus.cc:348] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bd28798e334ff18e9bf732ae428fa1" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43723 } } peers { permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 } } peers { permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 } }
I20260928 03:00:31.615517 24852 raft_consensus.cc:374] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:00:31.615599 24852 raft_consensus.cc:729] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c1359266ff4e558ed1263b72b3dd27, State: Initialized, Role: FOLLOWER
I20260928 03:00:31.616144 24852 consensus_queue.cc:260] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bd28798e334ff18e9bf732ae428fa1" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43723 } } peers { permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 } } peers { permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 } }
I20260928 03:00:31.616883 24854 raft_consensus.cc:348] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bd28798e334ff18e9bf732ae428fa1" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43723 } } peers { permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 } } peers { permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 } }
I20260928 03:00:31.617329 24854 raft_consensus.cc:374] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:00:31.617425 24854 raft_consensus.cc:729] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e006f07a51a34f9f8fabfe745853dcf3, State: Initialized, Role: FOLLOWER
I20260928 03:00:31.617861 24852 ts_tablet_manager.cc:1428] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260928 03:00:31.618005 24854 consensus_queue.cc:260] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bd28798e334ff18e9bf732ae428fa1" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43723 } } peers { permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 } } peers { permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 } }
I20260928 03:00:31.619262 24854 ts_tablet_manager.cc:1428] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260928 03:00:31.619556 24833 heartbeater.cc:503] Master 127.12.182.126:40713 was elected leader, sending a full tablet report...
W20260928 03:00:31.640789 24558 tablet.cc:2369] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:00:31.680279 24700 tablet.cc:2369] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:00:31.736888 24834 tablet.cc:2369] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:00:31.899044 24858 raft_consensus.cc:482] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:00:31.899408 24858 raft_consensus.cc:504] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bd28798e334ff18e9bf732ae428fa1" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43723 } } peers { permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 } } peers { permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 } }
I20260928 03:00:31.901046 24858 leader_election.cc:288] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6c1359266ff4e558ed1263b72b3dd27 (127.12.182.65:39715), e006f07a51a34f9f8fabfe745853dcf3 (127.12.182.67:42171)
I20260928 03:00:31.907232 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.901417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:48351 (local address 127.12.182.65:39715)
0928 03:00:31.901662 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:31.901669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:31.902324 (+ 655us) server_negotiation.cc:407] Connection header received
0928 03:00:31.902478 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:31.902481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:31.902539 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:31.902609 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:00:31.903215 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.903916 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.905662 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.906408 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.906554 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.906628 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:31.906879 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 03:00:31.907004 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 03:00:31.907325 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.901591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:48327 (local address 127.12.182.67:42171)
0928 03:00:31.901845 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:31.901851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:31.902271 (+ 420us) server_negotiation.cc:407] Connection header received
0928 03:00:31.902452 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:31.902455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:31.902510 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:31.902581 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:00:31.903167 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.903887 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.905688 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.906497 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.906640 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.906709 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:31.906988 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 03:00:31.907103 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 03:00:31.907430 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.901422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39715 (local address 127.12.182.66:48351)
0928 03:00:31.902219 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 03:00:31.902232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 03:00:31.902350 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:00:31.902637 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:00:31.902641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:00:31.902663 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:00:31.903067 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.903077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.904050 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.904054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.905496 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.905511 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.906600 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.906606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.906744 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.906766 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 03:00:31.906845 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 03:00:31.906961 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":23,"thread_start_us":391,"threads_started":1}
I20260928 03:00:31.907493 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:31.901321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:42171 (local address 127.12.182.66:48327)
0928 03:00:31.902171 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 03:00:31.902188 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:00:31.902332 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:00:31.902608 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:00:31.902612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:00:31.902634 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:00:31.903041 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.903053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.904004 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.904009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.905526 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:00:31.905538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:00:31.906699 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:00:31.906705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:00:31.906849 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:31.906875 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 03:00:31.906949 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 03:00:31.907063 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":97,"thread_start_us":680,"threads_started":1}
I20260928 03:00:31.908397 24788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca19b8971874374bd694693d5369d8e" candidate_uuid: "66bd28798e334ff18e9bf732ae428fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e006f07a51a34f9f8fabfe745853dcf3" is_pre_election: true
I20260928 03:00:31.908394 24511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca19b8971874374bd694693d5369d8e" candidate_uuid: "66bd28798e334ff18e9bf732ae428fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c1359266ff4e558ed1263b72b3dd27" is_pre_election: true
I20260928 03:00:31.908819 24511 raft_consensus.cc:2454] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66bd28798e334ff18e9bf732ae428fa1 in term 0.
I20260928 03:00:31.908819 24788 raft_consensus.cc:2454] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66bd28798e334ff18e9bf732ae428fa1 in term 0.
I20260928 03:00:31.909375 24590 leader_election.cc:302] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66bd28798e334ff18e9bf732ae428fa1, a6c1359266ff4e558ed1263b72b3dd27; no voters:
I20260928 03:00:31.909725 24858 raft_consensus.cc:2781] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:00:31.909860 24858 raft_consensus.cc:482] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:00:31.909958 24858 raft_consensus.cc:3037] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:00:31.912015 24858 raft_consensus.cc:504] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bd28798e334ff18e9bf732ae428fa1" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43723 } } peers { permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 } } peers { permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 } }
I20260928 03:00:31.912602 24858 leader_election.cc:288] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [CANDIDATE]: Term 1 election: Requested vote from peers a6c1359266ff4e558ed1263b72b3dd27 (127.12.182.65:39715), e006f07a51a34f9f8fabfe745853dcf3 (127.12.182.67:42171)
I20260928 03:00:31.912958 24511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca19b8971874374bd694693d5369d8e" candidate_uuid: "66bd28798e334ff18e9bf732ae428fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c1359266ff4e558ed1263b72b3dd27"
I20260928 03:00:31.913044 24788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca19b8971874374bd694693d5369d8e" candidate_uuid: "66bd28798e334ff18e9bf732ae428fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e006f07a51a34f9f8fabfe745853dcf3"
I20260928 03:00:31.913173 24511 raft_consensus.cc:3037] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:00:31.913203 24788 raft_consensus.cc:3037] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:00:31.916188 24511 raft_consensus.cc:2454] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66bd28798e334ff18e9bf732ae428fa1 in term 1.
I20260928 03:00:31.916200 24788 raft_consensus.cc:2454] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66bd28798e334ff18e9bf732ae428fa1 in term 1.
I20260928 03:00:31.916561 24590 leader_election.cc:302] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 66bd28798e334ff18e9bf732ae428fa1, a6c1359266ff4e558ed1263b72b3dd27, e006f07a51a34f9f8fabfe745853dcf3; no voters:
I20260928 03:00:31.916855 24858 raft_consensus.cc:2781] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:00:31.917274 24858 raft_consensus.cc:686] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 1 LEADER]: Becoming Leader. State: Replica: 66bd28798e334ff18e9bf732ae428fa1, State: Running, Role: LEADER
I20260928 03:00:31.917758 24858 consensus_queue.cc:237] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bd28798e334ff18e9bf732ae428fa1" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43723 } } peers { permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 } } peers { permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 } }
I20260928 03:00:31.922410 24343 catalog_manager.cc:5495] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66bd28798e334ff18e9bf732ae428fa1 (127.12.182.66). New cstate: current_term: 1 leader_uuid: "66bd28798e334ff18e9bf732ae428fa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bd28798e334ff18e9bf732ae428fa1" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 } health_report { overall_health: UNKNOWN } } }
I20260928 03:00:32.022004 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:32.016884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.12.182.66:43723)
0928 03:00:32.017075 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:32.017080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:32.017097 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:00:32.017224 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:32.017228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:32.017318 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:32.017478 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:00:32.017860 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:32.018581 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:32.019939 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:32.020367 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:32.020465 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:32.020875 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:00:32.021395 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:00:32.021498 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:32.021597 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 03:00:32.021700 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 03:00:32.029752 24511 raft_consensus.cc:1261] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27 [term 1 FOLLOWER]: Refusing update from remote peer 66bd28798e334ff18e9bf732ae428fa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:00:32.029752 24788 raft_consensus.cc:1261] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3 [term 1 FOLLOWER]: Refusing update from remote peer 66bd28798e334ff18e9bf732ae428fa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:00:32.030594 24863 consensus_queue.cc:1035] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6c1359266ff4e558ed1263b72b3dd27" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:00:32.030948 24858 consensus_queue.cc:1035] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e006f07a51a34f9f8fabfe745853dcf3" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:00:32.041322 24868 mvcc.cc:205] Tried to move back new op lower bound from 7334151913584050176 to 7334151913138782208. Current Snapshot: MvccSnapshot[applied={T|T < 7334151913584050176}]
I20260928 03:00:32.041978 24867 mvcc.cc:205] Tried to move back new op lower bound from 7334151913584050176 to 7334151913138782208. Current Snapshot: MvccSnapshot[applied={T|T < 7334151913584050176}]
I20260928 03:00:32.054155 24343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:00:32.055003 24343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:00:32.058792 24343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e148f0c3842f4d81b207458728686b1a: Sending DeleteTablet for 3 replicas of tablet 7ca19b8971874374bd694693d5369d8e
I20260928 03:00:32.060024 24768 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca19b8971874374bd694693d5369d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:00:32 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:33134
I20260928 03:00:32.060092 24634 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca19b8971874374bd694693d5369d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:00:32 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:54910
I20260928 03:00:32.060046 24767 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca19b8971874374bd694693d5369d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:00:32 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:33134
I20260928 03:00:32.061370 24877 tablet_replica.cc:330] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: stopping tablet replica
I20260928 03:00:32.062072 24877 raft_consensus.cc:2229] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:00:32.062252 24491 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca19b8971874374bd694693d5369d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:00:32 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:52874
I20260928 03:00:32.062690 24877 raft_consensus.cc:2258] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:00:32.062973 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 24423
I20260928 03:00:32.072428 24876 tablet_replica.cc:330] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: stopping tablet replica
I20260928 03:00:32.072863 24877 ts_tablet_manager.cc:1905] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:00:32.073055 24876 raft_consensus.cc:2229] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:00:32.073902 24876 raft_consensus.cc:2258] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:00:32.076097 24876 ts_tablet_manager.cc:1905] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:00:32.079635 24878 tablet_replica.cc:330] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: stopping tablet replica
I20260928 03:00:32.080168 24878 raft_consensus.cc:2229] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:00:32.080715 24878 raft_consensus.cc:2258] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:00:32.082842 24878 ts_tablet_manager.cc:1905] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:00:32.083132 24876 ts_tablet_manager.cc:1918] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:00:32.083341 24876 log.cc:1196] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/7ca19b8971874374bd694693d5369d8e
I20260928 03:00:32.083971 24876 ts_tablet_manager.cc:1939] T 7ca19b8971874374bd694693d5369d8e P 66bd28798e334ff18e9bf732ae428fa1: Deleting consensus metadata
Sep 28 03:00:32 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 03:00:32.085852 24328 catalog_manager.cc:4841] TS 66bd28798e334ff18e9bf732ae428fa1 (127.12.182.66:43723): tablet 7ca19b8971874374bd694693d5369d8e (table test-table [id=c3287c278a954f7c88cec5eff08bfb44]) successfully deleted
I20260928 03:00:32.086365 24877 ts_tablet_manager.cc:1918] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:00:32.086582 24877 log.cc:1196] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/7ca19b8971874374bd694693d5369d8e
I20260928 03:00:32.087122 24877 ts_tablet_manager.cc:1939] T 7ca19b8971874374bd694693d5369d8e P e006f07a51a34f9f8fabfe745853dcf3: Deleting consensus metadata
I20260928 03:00:32.088716 24328 catalog_manager.cc:4841] TS e006f07a51a34f9f8fabfe745853dcf3 (127.12.182.67:42171): tablet 7ca19b8971874374bd694693d5369d8e (table test-table [id=c3287c278a954f7c88cec5eff08bfb44]) successfully deleted
W20260928 03:00:32.089694 24328 catalog_manager.cc:4808] TS e006f07a51a34f9f8fabfe745853dcf3 (127.12.182.67:42171): delete failed for tablet 7ca19b8971874374bd694693d5369d8e because the tablet was not found. No further retry: Not found: Tablet not found: 7ca19b8971874374bd694693d5369d8e
I20260928 03:00:32.092808 24878 ts_tablet_manager.cc:1918] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:00:32.093052 24878 log.cc:1196] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/7ca19b8971874374bd694693d5369d8e
I20260928 03:00:32.093163 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:32.064224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.12.182.65:39715)
0928 03:00:32.064403 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:32.064410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:32.074549 (+ 10139us) server_negotiation.cc:407] Connection header received
0928 03:00:32.075026 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:32.075032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:32.075113 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:32.075187 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:00:32.076990 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:32.078093 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:32.079938 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:32.080620 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:32.080718 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:32.086117 (+ 5399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:00:32.086132 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:00:32.086148 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:00:32.086185 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:00:32.089826 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:32.090727 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:32.090740 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:32.090744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:32.090932 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:32.091372 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:32.091377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:32.091381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:32.091853 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:00:32.092168 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:32.092443 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 03:00:32.092697 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:00:32.093554 24878 ts_tablet_manager.cc:1939] T 7ca19b8971874374bd694693d5369d8e P a6c1359266ff4e558ed1263b72b3dd27: Deleting consensus metadata
I20260928 03:00:32.093796 24551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:00:32.186717 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 24423
I20260928 03:00:32.186694 24330 catalog_manager.cc:4841] TS a6c1359266ff4e558ed1263b72b3dd27 (127.12.182.65:39715): tablet 7ca19b8971874374bd694693d5369d8e (table test-table [id=c3287c278a954f7c88cec5eff08bfb44]) successfully deleted
I20260928 03:00:32.200706 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 24562
Sep 28 03:00:32 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 03:00:32.215713 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:32.201583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.12.182.66:43723)
0928 03:00:32.201783 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:32.201788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:32.201811 (+ 23us) server_negotiation.cc:407] Connection header received
0928 03:00:32.201876 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:32.201881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:32.201965 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:32.202050 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:00:32.203283 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:32.204094 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:32.205321 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:32.205775 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:32.205890 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:32.209924 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:00:32.209935 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:00:32.209951 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:00:32.209985 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:00:32.212796 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:32.213513 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:32.213521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:32.213525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:32.213862 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:32.214261 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:32.214269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:32.214273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:32.214608 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:00:32.214884 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:32.215166 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 03:00:32.215366 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 03:00:32.216367 24694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:00:32.327710 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 24562
I20260928 03:00:32.342137 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 24705
Sep 28 03:00:32 dist-test-slave-k5rg krb5kdc[23867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 03:00:32.356204 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:00:32.342864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.12.182.67:42171)
0928 03:00:32.343063 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 03:00:32.343068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:00:32.343087 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:00:32.343326 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:00:32.343332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:00:32.343397 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:00:32.343500 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:00:32.344742 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:32.345426 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:32.346681 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:00:32.347099 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:00:32.347198 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:00:32.351206 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:00:32.351220 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:00:32.351235 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:00:32.351267 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:00:32.353889 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:32.354593 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:32.354600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:32.354602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:32.354723 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:00:32.354998 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:00:32.355001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:00:32.355002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:00:32.355335 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:00:32.355531 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 03:00:32.355754 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 03:00:32.355902 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 03:00:32.356768 24828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:00:32.441915 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 24705
I20260928 03:00:32.454432 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 24311
I20260928 03:00:32.455480 24373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:00:32.563652 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 24311
I20260928 03:00:32.584542 13017 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 03:00:32.918942 13017 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 03:00:32.919130 13017 mini_ranger.cc:66] Stopping Ranger...
I20260928 03:00:33.254061 13017 mini_ranger.cc:68] Stopped Ranger
2026-09-28 03:00:33.254 UTC [23897] LOG: received smart shutdown request
2026-09-28 03:00:33.257 UTC [23897] LOG: background worker "logical replication launcher" (PID 23906) exited with exit code 1
2026-09-28 03:00:33.257 UTC [23901] LOG: shutting down
2026-09-28 03:00:33.403 UTC [23897] LOG: database system is shut down
2026-09-28T03:00:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47447 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:00:33 dist-test-slave-k5rg krb5kdc[24892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:00:33 dist-test-slave-k5rg krb5kdc[24892](info): set up 2 sockets
Sep 28 03:00:33 dist-test-slave-k5rg krb5kdc[24892](info): commencing operation
krb5kdc: starting...
W20260928 03:00:34.522004 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:00:34 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:00:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/rangeradmin_127.12.182.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/rangeradmin_127.12.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/rangerlookup_127.12.182.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/rangerlookup_127.12.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/HTTP_127.12.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/HTTP_127.12.182.84@KRBTEST.COM.keytab.
I20260928 03:00:34.832595 13017 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmphc7w/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/postgres -l logfile start
2026-09-28 03:00:37.177 UTC [24922] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 03:00:37.177 UTC [24922] LOG: listening on IPv4 address "127.12.182.84", port 34261
2026-09-28 03:00:37.180 UTC [24922] LOG: listening on Unix socket "/tmp/.s.PGSQL.34261"
2026-09-28 03:00:37.201 UTC [24925] LOG: database system was shut down at 2026-09-28 03:00:35 UTC
2026-09-28 03:00:37.205 UTC [24922] LOG: database system is ready to accept connections
I20260928 03:00:38.097929 13017 mini_postgres.cc:96] Postgres bound to 34261
2026-09-28 03:00:38.113 UTC [24933] FATAL: database "slave" does not exist
127.12.182.84:34261 - accepting connections
I20260928 03:00:38.113885 13017 mini_ranger.cc:161] Starting Ranger...
I20260928 03:00:38.137732 13017 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 03:00:38.597121 13017 mini_ranger.cc:87] Created ranger Postgres database
I20260928 03:00:38.597229 13017 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-admin
2026-09-28 03:00:38,861 [I] DB FLAVOR :POSTGRES
2026-09-28 03:00:38,861 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 03:00:38,861 [I] Checking connection..
2026-09-28 03:00:38,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 03:00:39,112 [I] Checking connection passed.
2026-09-28 03:00:39,113 [I] --------- Verifying version history table ---------
2026-09-28 03:00:39,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 03:00:39,340 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 03:00:39,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 03:00:39,579 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 03:00:39,579 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 03:00:39,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 03:00:39.818 UTC [24997] WARNING: there is no transaction in progress
2026-09-28 03:00:39,824 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 03:00:39,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 03:00:40,080 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 03:00:40,080 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 03:00:40,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 03:00:40,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 03:00:40,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k5rg.c.gcp-upstream.internal','N') ;"
2026-09-28 03:00:40,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 03:00:41,041 [I] Table x_portal_user does not exist in database ranger
2026-09-28 03:00:41,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 03:00:41,282 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 03:00:41,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 03:00:41,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 03:00:41,762 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 03:00:41,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 03:00:42.693 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.710 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.728 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.744 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.760 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.777 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.830 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.836 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.850 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.859 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.868 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.881 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.891 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.897 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.903 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:42.910 UTC [25105] WARNING: there is no transaction in progress
2026-09-28 03:00:43,502 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 03:00:43,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 03:00:43,736 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 03:00:43,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 03:00:43,979 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 03:00:43,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 03:00:44,217 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 03:00:44,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 03:00:44,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 03:00:44,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k5rg.c.gcp-upstream.internal';"
2026-09-28 03:00:44,870 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 03:00:44,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 03:00:45,115 [I] DB_PATCHES have already been applied
I20260928 03:00:45.121331 13017 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmphc7w/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 03:00:45.121416 13017 mini_ranger.cc:193] Using host: 127.12.182.84
I20260928 03:00:45.134567 13017 mini_ranger.cc:227] Ranger admin URL: http://127.12.182.84:37535
Sep 28, 2026 3:00:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:00:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:00:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:00:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:00:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:00:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:00:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 03:00:46.159924 13017 mini_ranger.cc:160] Time spent starting Ranger: real 8.046s user 0.002s sys 0.033s
Sep 28, 2026 3:01:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:01:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 3:01:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:01:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:01:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 03:01:07.241397 13017 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/rangerkms_127.12.182.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/rangerkms_127.12.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.182.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.182.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.182.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/HTTP_127.12.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.182.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/HTTP_127.12.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 03:01:07 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564467, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 03:01:07.392498 13017 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 03:01:07.411257 13017 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 03:01:08.199224 13017 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 03:01:08.199333 13017 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-kms
I20260928 03:01:08.199393 13017 mini_ranger_kms.cc:227] Using postgres at 127.12.182.84:34261
2026-09-28 03:01:08,524 [I] DB FLAVOR :POSTGRES
2026-09-28 03:01:08,524 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 03:01:08,524 [I] Checking connection
2026-09-28 03:01:08,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 03:01:08,789 [I] connection success
2026-09-28 03:01:08,789 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 03:01:08,789 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 03:01:08,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 03:01:09,067 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 03:01:09,067 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 03:01:09,067 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 03:01:09,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.182.84:34261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 03:01:09,327 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 03:01:09.761855 13017 mini_ranger_kms.cc:310] Created kms service
I20260928 03:01:09.941566 13017 mini_ranger_kms.cc:326] Created kudu user
I20260928 03:01:09.983018 13017 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 03:01:09 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564467, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:01:10.364524 13017 mini_ranger_kms.cc:384] Added ranger policy
I20260928 03:01:10.364853 13017 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmphc7w/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmphc7w/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 03:01:10.364915 13017 mini_ranger_kms.cc:240] Using host: 127.12.182.84
I20260928 03:01:10.377794 13017 mini_ranger_kms.cc:276] Ranger KMS PID: 25285
I20260928 03:01:10.377912 13017 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.182.84:33267
Sep 28, 2026 3:01:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:01:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:01:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:01:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:01:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:01:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 03:01:12.437114 13017 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.045s user 0.049s sys 0.286s
I20260928 03:01:12.437376 13017 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 03:01:12.437561 13017 mini_ranger_kms.cc:401] 127.12.182.84:33267/kms/v1/keys
Sep 28 03:01:12 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564472, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.182.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 03:01:12 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564472, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.182.84@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
Sep 28 03:01:13 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564472, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.182.84@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
Sep 28 03:01:15 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:01:15.476974 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:46449
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:46449
--ranger_config_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:33267/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:15.786126 25336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:15.786396 25336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:15.786520 25336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:15.793687 25336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:01:15.793779 25336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:15.793831 25336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:15.793907 25336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:01:15.793968 25336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:01:15.803519 25336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.182.84:33267/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:46449
--ranger_config_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:46449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:15.804657 25336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:15.805963 25336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:15.816362 25341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:15.816402 25342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:15.816529 25344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:15.817099 25336 server_base.cc:1034] running on GCE node
I20260928 03:01:15.817860 25336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:15.819135 25336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:15.820333 25336 hybrid_clock.cc:648] HybridClock initialized: now 1790564475820280 us; error 44 us; skew 500 ppm
I20260928 03:01:15.820639 25336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:15 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564475, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:15.828658 25336 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:01:15.830071 25336 webserver.cc:460] Webserver started at http://127.12.182.126:34551/ using document root <none> and password file <none>
I20260928 03:01:15.830685 25336 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:15.830830 25336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:15.831115 25336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:01:15 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564475, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:01:15.886791 25336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "a3ea60c0e12c4e118bec23ccc7321aa5"
format_stamp: "Formatted at 2026-09-28 03:01:15 on dist-test-slave-k5rg"
server_key: "7a71e3a3251d809e060da5ca6e9edd0b"
server_key_iv: "ff5886186769bec216c792ac2fcf92b4"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:15.887918 25336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "a3ea60c0e12c4e118bec23ccc7321aa5"
format_stamp: "Formatted at 2026-09-28 03:01:15 on dist-test-slave-k5rg"
server_key: "7a71e3a3251d809e060da5ca6e9edd0b"
server_key_iv: "ff5886186769bec216c792ac2fcf92b4"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:15.894767 25336 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 03:01:15.913774 25351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:15.915452 25336 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:01:15.915652 25336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "a3ea60c0e12c4e118bec23ccc7321aa5"
format_stamp: "Formatted at 2026-09-28 03:01:15 on dist-test-slave-k5rg"
server_key: "7a71e3a3251d809e060da5ca6e9edd0b"
server_key_iv: "ff5886186769bec216c792ac2fcf92b4"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:15.915894 25336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:15.936240 25336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:15.940377 25336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:15.940619 25336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:15.957590 25336 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:46449
I20260928 03:01:15.957638 25402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:46449 every 8 connection(s)
I20260928 03:01:15.958808 25336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:01:15.962294 25403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:15.966645 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 25336
I20260928 03:01:15.966841 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:01:15.967765 13017 external_mini_cluster.cc:1428] Setting key 505bc9890f37aab42c278fe044b4f721
I20260928 03:01:15.973130 25403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: Bootstrap starting.
I20260928 03:01:15.976505 25403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:15.977686 25403 log.cc:824] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:15.980738 25403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: No bootstrap required, opened a new log
Sep 28 03:01:15 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:15.990608 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:15.969553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.12.182.126:46449)
0928 03:01:15.970325 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:15.970335 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:15.970388 (+ 53us) server_negotiation.cc:407] Connection header received
0928 03:01:15.972106 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:15.972128 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:15.972491 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:15.972776 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:15.974376 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:15.975634 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:15.977385 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:15.978147 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:15.978324 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:15.982558 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:15.982569 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:15.982587 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:15.982641 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:15.986408 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:15.987054 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:15.987062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:15.987073 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:15.987250 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:15.987593 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:15.987598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:15.987602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:15.988277 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:15.988582 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:15.988930 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 03:01:15.989321 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":350,"threads_started":1}
I20260928 03:01:15.990533 25403 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } }
I20260928 03:01:15.990962 25403 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:15.991073 25403 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ea60c0e12c4e118bec23ccc7321aa5, State: Initialized, Role: FOLLOWER
I20260928 03:01:15.991698 25403 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } }
I20260928 03:01:15.991959 25403 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:01:15.992113 25403 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:01:15.992264 25403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:15.994613 25403 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } }
I20260928 03:01:15.995067 25403 leader_election.cc:302] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3ea60c0e12c4e118bec23ccc7321aa5; no voters:
I20260928 03:01:15.995589 25403 leader_election.cc:288] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:01:15.995735 25408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:01:15.996335 25408 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 1 LEADER]: Becoming Leader. State: Replica: a3ea60c0e12c4e118bec23ccc7321aa5, State: Running, Role: LEADER
I20260928 03:01:15.996837 25408 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } }
I20260928 03:01:15.997493 25403 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:01:15.999990 25410 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3ea60c0e12c4e118bec23ccc7321aa5. Latest consensus state: current_term: 1 leader_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } } }
I20260928 03:01:16.000406 25410 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:16.001051 25409 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } } }
I20260928 03:01:16.001330 25409 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:16.006568 25417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:01:16.007063 25403 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 03:01:16.010768 25417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:01:16.036533 25417 catalog_manager.cc:1347] Generated new cluster ID: 420acb94a828464b97bdac3def23f10e
I20260928 03:01:16.036677 25417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:01:16.055197 25417 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:01:16.057051 25417 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:01:16.068348 25417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: Generated new TSK 0
I20260928 03:01:16.069415 25417 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 03:01:16.780068 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:33267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46449
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:01:17 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564477, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 03:01:17.158011 25448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:17.158384 25448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:17.158550 25448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:17.166106 25448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:17.166218 25448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:17.166393 25448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:01:17.176479 25448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.182.84:33267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:17.177915 25448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:17.179284 25448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:17.194176 25458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:17.202562 25461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:17.202704 25448 server_base.cc:1034] running on GCE node
W20260928 03:01:17.203016 25459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:17.203704 25448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:17.204754 25448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:17.205998 25448 hybrid_clock.cc:648] HybridClock initialized: now 1790564477205937 us; error 54 us; skew 500 ppm
I20260928 03:01:17.206295 25448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:17 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564477, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:17.209194 25448 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:01:17.213387 25448 webserver.cc:460] Webserver started at http://127.12.182.65:42151/ using document root <none> and password file <none>
I20260928 03:01:17.214382 25448 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:17.214576 25448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:17.214972 25448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:01:17 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564477, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:01:17.236824 25448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "3718c60dd92b48bcbf78d9b9e5f0d412"
format_stamp: "Formatted at 2026-09-28 03:01:17 on dist-test-slave-k5rg"
server_key: "2b3733a24683fb25d0dcba8233def03e"
server_key_iv: "a0696b842192ecce43c9162f1b1f7e02"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:17.240795 25448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "3718c60dd92b48bcbf78d9b9e5f0d412"
format_stamp: "Formatted at 2026-09-28 03:01:17 on dist-test-slave-k5rg"
server_key: "2b3733a24683fb25d0dcba8233def03e"
server_key_iv: "a0696b842192ecce43c9162f1b1f7e02"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:17.253520 25448 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.011s sys 0.000s
I20260928 03:01:17.279783 25468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:17.281761 25448 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.008s
I20260928 03:01:17.282013 25448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "3718c60dd92b48bcbf78d9b9e5f0d412"
format_stamp: "Formatted at 2026-09-28 03:01:17 on dist-test-slave-k5rg"
server_key: "2b3733a24683fb25d0dcba8233def03e"
server_key_iv: "a0696b842192ecce43c9162f1b1f7e02"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:17.282359 25448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:17.331290 25448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:17.336323 25448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:17.336699 25448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:17.342288 25448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:17.344131 25448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:17.344234 25448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:17.344408 25448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:17.344472 25448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:17.417528 25448 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:37655
I20260928 03:01:17.417544 25580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:37655 every 8 connection(s)
I20260928 03:01:17.419061 25448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:01:17.420938 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 25448
I20260928 03:01:17.421146 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 03:01:17.421973 13017 external_mini_cluster.cc:1428] Setting key 011d19886ca9d10ffaf690a819f4da14
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
Sep 28 03:01:17 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564477, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:17.476933 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:17.425848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:58773 (local address 127.12.182.126:46449)
0928 03:01:17.452323 (+ 26475us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:17.452328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:17.457342 (+ 5014us) server_negotiation.cc:407] Connection header received
0928 03:01:17.457420 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:17.457425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:17.457524 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:17.457651 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:17.460581 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:17.461720 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:17.464098 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:17.464774 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:17.464873 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:17.469464 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:17.469471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:17.469475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:17.469517 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:17.473060 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:17.474079 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:17.474085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:17.474089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:17.474246 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:17.474801 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:17.474805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:17.474809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:17.475168 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:17.475315 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:17.476310 (+ 995us) server_negotiation.cc:299] Negotiation successful
0928 03:01:17.476545 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26352,"thread_start_us":25817,"threads_started":1}
I20260928 03:01:17.479521 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:17.425248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46449 (local address 127.12.182.65:58773)
0928 03:01:17.454648 (+ 29400us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:17.454701 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:17.456934 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:17.458994 (+ 2060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:17.459006 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:17.459592 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:17.460372 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:17.460398 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:17.461932 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:17.461937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:17.463922 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:17.463939 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:17.465203 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:17.465209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:17.465494 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:17.466257 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:17.466287 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:17.469162 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:17.473264 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:17.473273 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:17.473292 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:17.473925 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:17.474409 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:17.474413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:17.474416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:17.474672 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:17.475408 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:17.475416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:17.475862 (+ 446us) client_negotiation.cc:769] Sending connection context
0928 03:01:17.477516 (+ 1654us) client_negotiation.cc:240] Negotiation successful
0928 03:01:17.478109 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28107,"thread_start_us":28244,"threads_started":1}
I20260928 03:01:17.482321 25581 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46449
I20260928 03:01:17.482641 25581 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:17.483439 25581 heartbeater.cc:511] Master 127.12.182.126:46449 requested a full tablet report, sending...
I20260928 03:01:17.486145 25368 ts_manager.cc:194] Registered new tserver with Master: 3718c60dd92b48bcbf78d9b9e5f0d412 (127.12.182.65:37655)
I20260928 03:01:17.488170 25368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:58773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 03:01:17.489969 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:33267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46449
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:01:17 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564477, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
Sep 28 03:01:17 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564477, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
W20260928 03:01:17.982283 25587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:17.982762 25587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:17.982971 25587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:17.994127 25587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:17.994329 25587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:17.994556 25587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:01:18.009814 25587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.182.84:33267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:18.011751 25587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:18.013885 25587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:18.046552 25595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:18.056354 25587 server_base.cc:1034] running on GCE node
W20260928 03:01:18.056833 25594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:18.057461 25597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:18.058346 25587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:18.065126 25587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:18.066382 25587 hybrid_clock.cc:648] HybridClock initialized: now 1790564478066274 us; error 105 us; skew 500 ppm
I20260928 03:01:18.066803 25587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:18 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564478, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:18.071012 25587 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:01:18.074922 25587 webserver.cc:460] Webserver started at http://127.12.182.66:42271/ using document root <none> and password file <none>
I20260928 03:01:18.075858 25587 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:18.076063 25587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:18.076455 25587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:01:18 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564478, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:01:18.118270 25587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "f55efc7b9ac84cca930b34e5f4340d12"
format_stamp: "Formatted at 2026-09-28 03:01:18 on dist-test-slave-k5rg"
server_key: "06fcabdbb520b0f4e0f596d47d04eb3f"
server_key_iv: "c586ff48e985ba0d6d57232c3dd4a0e3"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:18.119977 25587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "f55efc7b9ac84cca930b34e5f4340d12"
format_stamp: "Formatted at 2026-09-28 03:01:18 on dist-test-slave-k5rg"
server_key: "06fcabdbb520b0f4e0f596d47d04eb3f"
server_key_iv: "c586ff48e985ba0d6d57232c3dd4a0e3"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:18.140265 25587 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.003s sys 0.016s
I20260928 03:01:18.188995 25604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:18.194995 25587 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.004s sys 0.002s
I20260928 03:01:18.195326 25587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "f55efc7b9ac84cca930b34e5f4340d12"
format_stamp: "Formatted at 2026-09-28 03:01:18 on dist-test-slave-k5rg"
server_key: "06fcabdbb520b0f4e0f596d47d04eb3f"
server_key_iv: "c586ff48e985ba0d6d57232c3dd4a0e3"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:18.195721 25587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:18.266280 25587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:18.271627 25587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:18.272013 25587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:18.273257 25587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:18.275192 25587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:18.275355 25587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:18.275525 25587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:18.275630 25587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:18.313364 25587 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:41963
I20260928 03:01:18.313405 25723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:41963 every 8 connection(s)
I20260928 03:01:18.315165 25587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:01:18.316250 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 25587
I20260928 03:01:18.316423 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 03:01:18.317020 13017 external_mini_cluster.cc:1428] Setting key 2cd681f19f0a9adecadfbcfe572ec115
Sep 28 03:01:18 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564478, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
I20260928 03:01:18.360961 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:18.319791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:34741 (local address 127.12.182.126:46449)
0928 03:01:18.338672 (+ 18881us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:18.338680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:18.339911 (+ 1231us) server_negotiation.cc:407] Connection header received
0928 03:01:18.342445 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:18.342452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:18.342548 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:18.342626 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:18.344978 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:18.346145 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:18.348493 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:18.349070 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:18.349188 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:18.353445 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:18.353453 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:18.353458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:18.353500 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:18.357034 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:18.358020 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:18.358026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:18.358030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:18.358174 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:18.358680 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:18.358684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:18.358687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:18.359022 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:18.359153 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:18.360216 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0928 03:01:18.360529 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18730,"thread_start_us":18886,"threads_started":1}
I20260928 03:01:18.361999 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:18.320450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46449 (local address 127.12.182.66:34741)
0928 03:01:18.339367 (+ 18917us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:18.339426 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:18.342046 (+ 2620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:18.343239 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:18.343252 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:18.343930 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:18.344755 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:18.344783 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:18.346334 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:18.346340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:18.348346 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:18.348363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:18.349308 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:18.349315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:18.349594 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:18.350441 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:18.350477 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:18.353177 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:18.357217 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:18.357227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:18.357246 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:18.357865 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:18.358333 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:18.358337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:18.358340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:18.358563 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:18.359245 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:18.359265 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:18.359709 (+ 444us) client_negotiation.cc:769] Sending connection context
0928 03:01:18.360038 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 03:01:18.360642 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18575,"thread_start_us":18376,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
I20260928 03:01:18.363538 25724 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46449
I20260928 03:01:18.363900 25724 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:18.364604 25724 heartbeater.cc:511] Master 127.12.182.126:46449 requested a full tablet report, sending...
I20260928 03:01:18.366084 25368 ts_manager.cc:194] Registered new tserver with Master: f55efc7b9ac84cca930b34e5f4340d12 (127.12.182.66:41963)
I20260928 03:01:18.366876 25368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:34741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 03:01:18.397089 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:33267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46449
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:01:18.491720 25581 heartbeater.cc:503] Master 127.12.182.126:46449 was elected leader, sending a full tablet report...
W20260928 03:01:18.724213 25730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:18.724483 25730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:18.724599 25730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:18.731977 25730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:18.732096 25730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:18.732268 25730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:01:18.742571 25730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.182.84:33267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:18.743837 25730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:18.745040 25730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:18.756613 25735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:18.757026 25736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:18.757372 25738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:18.758585 25730 server_base.cc:1034] running on GCE node
I20260928 03:01:18.759109 25730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:18.759958 25730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:18.761142 25730 hybrid_clock.cc:648] HybridClock initialized: now 1790564478761090 us; error 42 us; skew 500 ppm
I20260928 03:01:18.761426 25730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:18 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564478, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:18.764357 25730 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:01:18.765829 25730 webserver.cc:460] Webserver started at http://127.12.182.67:44951/ using document root <none> and password file <none>
I20260928 03:01:18.766474 25730 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:18.766574 25730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:18.766853 25730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:01:18 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564478, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:01:18.780388 25730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "99f06afcebcd400eb9cb6c9b7704c0f0"
format_stamp: "Formatted at 2026-09-28 03:01:18 on dist-test-slave-k5rg"
server_key: "be99b2a2b7756c9db3881294d916e849"
server_key_iv: "2e6b63faee62cb342604043258022032"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:18.781334 25730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "99f06afcebcd400eb9cb6c9b7704c0f0"
format_stamp: "Formatted at 2026-09-28 03:01:18 on dist-test-slave-k5rg"
server_key: "be99b2a2b7756c9db3881294d916e849"
server_key_iv: "2e6b63faee62cb342604043258022032"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:18.786273 25730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 03:01:18.806739 25745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:18.808285 25730 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 03:01:18.808451 25730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "99f06afcebcd400eb9cb6c9b7704c0f0"
format_stamp: "Formatted at 2026-09-28 03:01:18 on dist-test-slave-k5rg"
server_key: "be99b2a2b7756c9db3881294d916e849"
server_key_iv: "2e6b63faee62cb342604043258022032"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:18.808706 25730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:18.829216 25730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:18.833030 25730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:18.833299 25730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:18.834211 25730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:18.835554 25730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:18.835628 25730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:18.835737 25730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:18.835795 25730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:18.863276 25730 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:40561
I20260928 03:01:18.863296 25857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:40561 every 8 connection(s)
I20260928 03:01:18.864468 25730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:01:18.871261 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 25730
I20260928 03:01:18.871452 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 03:01:18.871873 13017 external_mini_cluster.cc:1428] Setting key 94b398889d5f46b799a238bef33cc263
Sep 28 03:01:18 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564478, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:18.882553 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:18.866928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:45703 (local address 127.12.182.126:46449)
0928 03:01:18.867344 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:18.867349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:18.868251 (+ 902us) server_negotiation.cc:407] Connection header received
0928 03:01:18.869951 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:18.869956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:18.870020 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:18.870111 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:18.871535 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:18.872276 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:18.874088 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:18.874559 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:18.874651 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:18.877705 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:18.877709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:18.877713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:18.877753 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:18.880107 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:18.880721 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:18.880725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:18.880727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:18.880861 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:18.881202 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:18.881205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:18.881207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:18.881451 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:18.881555 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:18.882167 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 03:01:18.882314 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":202,"threads_started":1}
I20260928 03:01:18.883473 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:18.867447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46449 (local address 127.12.182.67:45703)
0928 03:01:18.868103 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:18.868139 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:18.869711 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:18.870299 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:18.870306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:18.870687 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:18.871367 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:18.871393 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:18.872410 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:18.872414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:18.873956 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:18.873970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:18.874754 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:18.874761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:18.874963 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:18.875557 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:18.875579 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:18.877497 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:18.880227 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:18.880233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:18.880244 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:18.880634 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:18.880941 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:18.880944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:18.880946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:18.881128 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:18.881534 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:18.881540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:18.881885 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 03:01:18.882094 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 03:01:18.882430 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":282,"threads_started":1}
I20260928 03:01:18.884785 25858 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46449
I20260928 03:01:18.885116 25858 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:18.885908 25858 heartbeater.cc:511] Master 127.12.182.126:46449 requested a full tablet report, sending...
I20260928 03:01:18.887372 25368 ts_manager.cc:194] Registered new tserver with Master: 99f06afcebcd400eb9cb6c9b7704c0f0 (127.12.182.67:40561)
I20260928 03:01:18.888266 25368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:45703
I20260928 03:01:18.899225 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:01:18.900861 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 25448
Sep 28 03:01:18 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 03:01:18.915997 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:18.901739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.12.182.65:37655)
0928 03:01:18.902365 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:18.902372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:18.902396 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:01:18.902448 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:18.902453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:18.902677 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:18.902850 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:18.904238 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:18.905209 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:18.906702 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:18.907252 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:18.907355 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:18.911043 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:18.911054 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:18.911069 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:18.911103 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:18.913723 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:18.914416 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:18.914422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:18.914424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:18.914564 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:18.914824 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:18.914827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:18.914829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:18.915131 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:18.915351 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:18.915545 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 03:01:18.915731 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":317,"threads_started":1}
I20260928 03:01:18.917232 25576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:18.997581 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 25448
I20260928 03:01:19.010376 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 25587
Sep 28 03:01:19 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 03:01:19.025938 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:19.011245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.12.182.66:41963)
0928 03:01:19.011761 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:19.011768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:19.011793 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:01:19.011863 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:19.011869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:19.012065 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:19.012197 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:19.013494 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:19.014663 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:19.016315 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:19.016873 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:19.016987 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:19.020964 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:19.020975 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:19.020989 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:19.021028 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:19.023657 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:19.024252 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:19.024259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:19.024261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:19.024397 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:19.024673 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:19.024676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:19.024678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:19.024997 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:19.025198 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:19.025384 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 03:01:19.025574 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":263,"threads_started":1}
I20260928 03:01:19.027112 25719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:19.107720 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 25587
I20260928 03:01:19.119625 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 25730
Sep 28 03:01:19 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 03:01:19.135309 25867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:19.120581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.12.182.67:40561)
0928 03:01:19.121289 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:19.121296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:19.121323 (+ 27us) server_negotiation.cc:407] Connection header received
0928 03:01:19.121378 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:19.121386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:19.121682 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:19.121820 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:19.123095 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:19.124087 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:19.125353 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:19.125930 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:19.126041 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:19.130179 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:19.130190 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:19.130203 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:19.130236 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:19.132697 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:19.133262 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:19.133267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:19.133270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:19.133390 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:19.133807 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:19.133811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:19.133813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:19.134157 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:19.134469 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:19.134737 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 03:01:19.134990 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":442,"threads_started":1}
I20260928 03:01:19.136480 25853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:19.219295 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 25730
I20260928 03:01:19.231472 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 25336
I20260928 03:01:19.232664 25398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:19.336417 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 25336
I20260928 03:01:19.353233 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:46449
--webserver_interface=127.12.182.126
--webserver_port=34551
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:46449
--ranger_config_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:33267/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.182.84:33267/kms with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:19.657531 25870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:19.657829 25870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:19.657958 25870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:19.664963 25870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:01:19.665071 25870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:19.665149 25870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:19.665225 25870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:01:19.665318 25870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:01:19.675055 25870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.182.84:33267/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:46449
--ranger_config_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:46449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=34551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:19.676287 25870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:19.677521 25870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:19.690006 25875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:19.689929 25876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:19.689903 25878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:19.690623 25870 server_base.cc:1034] running on GCE node
I20260928 03:01:19.691342 25870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:19.692651 25870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:19.693843 25870 hybrid_clock.cc:648] HybridClock initialized: now 1790564479693798 us; error 33 us; skew 500 ppm
I20260928 03:01:19.694173 25870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:19 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564479, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:19.696960 25870 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:01:19.698342 25870 webserver.cc:460] Webserver started at http://127.12.182.126:34551/ using document root <none> and password file <none>
I20260928 03:01:19.698954 25870 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:19.699069 25870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 03:01:19 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564479, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:01:19.756675 25870 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 03:01:19.760277 25886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:19.761886 25870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:01:19.762077 25870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "a3ea60c0e12c4e118bec23ccc7321aa5"
format_stamp: "Formatted at 2026-09-28 03:01:15 on dist-test-slave-k5rg"
server_key: "7a71e3a3251d809e060da5ca6e9edd0b"
server_key_iv: "ff5886186769bec216c792ac2fcf92b4"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:19.763353 25870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:19.780735 25870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:19.784354 25870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:19.784575 25870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:19.801273 25870 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:46449
I20260928 03:01:19.801297 25937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:46449 every 8 connection(s)
I20260928 03:01:19.802697 25870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:01:19.809032 25938 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:01:19.812048 25938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: Bootstrap starting.
I20260928 03:01:19.812702 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 25870
I20260928 03:01:19.813499 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:33267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:37655
--local_ip_for_outbound_sockets=127.12.182.65
--tserver_master_addrs=127.12.182.126:46449
--webserver_port=42151
--webserver_interface=127.12.182.65
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.182.84:33267/kms with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:01:19.830075 25938 log.cc:824] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:19.838384 25938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:01:19.838824 25938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: Bootstrap complete.
I20260928 03:01:19.847236 25938 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } }
I20260928 03:01:19.848019 25938 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ea60c0e12c4e118bec23ccc7321aa5, State: Initialized, Role: FOLLOWER
I20260928 03:01:19.848528 25938 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } }
I20260928 03:01:19.848726 25938 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:01:19.848822 25938 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:01:19.848970 25938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 03:01:19.852623 25938 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } }
I20260928 03:01:19.853060 25938 leader_election.cc:302] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3ea60c0e12c4e118bec23ccc7321aa5; no voters:
I20260928 03:01:19.853582 25938 leader_election.cc:288] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 03:01:19.853785 25942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 03:01:19.854643 25942 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [term 2 LEADER]: Becoming Leader. State: Replica: a3ea60c0e12c4e118bec23ccc7321aa5, State: Running, Role: LEADER
I20260928 03:01:19.855247 25942 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } }
I20260928 03:01:19.855423 25938 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:01:19.857662 25943 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } } }
I20260928 03:01:19.858038 25943 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:19.858570 25944 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3ea60c0e12c4e118bec23ccc7321aa5. Latest consensus state: current_term: 2 leader_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3ea60c0e12c4e118bec23ccc7321aa5" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46449 } } }
I20260928 03:01:19.859082 25944 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:19.860193 25948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:01:19.866315 25948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:01:19.867746 25948 catalog_manager.cc:1259] Loaded cluster ID: 420acb94a828464b97bdac3def23f10e
I20260928 03:01:19.867892 25948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:01:19.872002 25948 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:01:19.873751 25948 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: Loaded TSK: 0
I20260928 03:01:19.875389 25948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:01:19.877779 25938 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 03:01:20.238829 25940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:20.239338 25940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:20.239543 25940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:20.250732 25940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:20.250943 25940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:20.251180 25940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:01:20.266498 25940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.182.84:33267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:37655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=42151
--tserver_master_addrs=127.12.182.126:46449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:20.268667 25940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:20.270537 25940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:20.320750 25975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:20.324976 25976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:20.333426 25978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:20.341229 25940 server_base.cc:1034] running on GCE node
I20260928 03:01:20.342290 25940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:20.345044 25940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:20.346357 25940 hybrid_clock.cc:648] HybridClock initialized: now 1790564480346292 us; error 60 us; skew 500 ppm
I20260928 03:01:20.346797 25940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:20 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:20.350988 25940 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:01:20.353824 25940 webserver.cc:460] Webserver started at http://127.12.182.65:42151/ using document root <none> and password file <none>
I20260928 03:01:20.354871 25940 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:20.355101 25940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 03:01:20 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:01:20.388303 25940 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.002s sys 0.001s
I20260928 03:01:20.395784 25987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:20.397228 25940 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260928 03:01:20.397405 25940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "3718c60dd92b48bcbf78d9b9e5f0d412"
format_stamp: "Formatted at 2026-09-28 03:01:17 on dist-test-slave-k5rg"
server_key: "2b3733a24683fb25d0dcba8233def03e"
server_key_iv: "a0696b842192ecce43c9162f1b1f7e02"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:20.398487 25940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:20.447422 25940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:20.451591 25940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:20.451857 25940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:20.452955 25940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:20.454665 25940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:20.454767 25940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:20.454870 25940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:20.454952 25940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:20.507537 25940 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:37655
I20260928 03:01:20.507591 26099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:37655 every 8 connection(s)
I20260928 03:01:20.508874 25940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:01:20.512918 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 25940
I20260928 03:01:20.513756 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:33267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:41963
--local_ip_for_outbound_sockets=127.12.182.66
--tserver_master_addrs=127.12.182.126:46449
--webserver_port=42271
--webserver_interface=127.12.182.66
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.182.84:33267/kms with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:01:20 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:20.540979 26103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:20.512049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46449 (local address 127.12.182.65:34011)
0928 03:01:20.516275 (+ 4226us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:20.516319 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:20.518387 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:20.520722 (+ 2335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:20.520731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:20.521751 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:20.522485 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:20.522510 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:20.524822 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:20.524828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:20.526778 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:20.526803 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:20.528265 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:20.528272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:20.528538 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:20.529264 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:20.529291 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:20.531902 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:20.535956 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:20.535965 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:20.535981 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:20.536552 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:20.537076 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:20.537081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:20.537085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:20.537343 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:20.538354 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:20.538362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:20.538771 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 03:01:20.539048 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 03:01:20.539602 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3971,"thread_start_us":3777,"threads_started":1}
I20260928 03:01:20.544270 26102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:20.511858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:34011 (local address 127.12.182.126:46449)
0928 03:01:20.512514 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:20.512527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:20.516455 (+ 3928us) server_negotiation.cc:407] Connection header received
0928 03:01:20.519071 (+ 2616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:20.519092 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:20.519565 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:20.519936 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:20.522715 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:20.524639 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:20.526997 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:20.528020 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:20.528994 (+ 974us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:20.532211 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:20.532219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:20.532236 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:20.532278 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:20.535788 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:20.536740 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:20.536749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:20.536757 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:20.536943 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:20.537504 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:20.537509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:20.537513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:20.538195 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:20.542263 (+ 4068us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:20.542620 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 03:01:20.543029 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":232,"threads_started":1}
I20260928 03:01:20.547740 26100 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46449
I20260928 03:01:20.548027 26100 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:20.548821 26100 heartbeater.cc:511] Master 127.12.182.126:46449 requested a full tablet report, sending...
W20260928 03:01:20.551867 26100 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:46449 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 03:01:20.552073 26100 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:20.552307 26100 heartbeater.cc:511] Master 127.12.182.126:46449 requested a full tablet report, sending...
I20260928 03:01:20.553473 26100 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:20.553802 26100 heartbeater.cc:511] Master 127.12.182.126:46449 requested a full tablet report, sending...
W20260928 03:01:20.554775 26100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 03:01:20.908891 26104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:20.909307 26104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:20.909515 26104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:20.921096 26104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:20.921275 26104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:20.921499 26104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:01:20.936888 26104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.182.84:33267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:41963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=42271
--tserver_master_addrs=127.12.182.126:46449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:20.938750 26104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:20.940462 26104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:20.965260 26120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:20.966346 26121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:20.973824 26124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:20.974959 26104 server_base.cc:1034] running on GCE node
I20260928 03:01:20.975679 26104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:20.977873 26104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:20.979173 26104 hybrid_clock.cc:648] HybridClock initialized: now 1790564480979099 us; error 61 us; skew 500 ppm
I20260928 03:01:20.979553 26104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:20 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:20.987774 26104 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:01:20.989753 26104 webserver.cc:460] Webserver started at http://127.12.182.66:42271/ using document root <none> and password file <none>
I20260928 03:01:20.990636 26104 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:20.990783 26104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 03:01:20 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
Sep 28 03:01:21 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564481, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:21.029305 26104 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.007s sys 0.002s
I20260928 03:01:21.042254 26131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:21.045213 26104 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.010s
I20260928 03:01:21.045398 26104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "f55efc7b9ac84cca930b34e5f4340d12"
format_stamp: "Formatted at 2026-09-28 03:01:18 on dist-test-slave-k5rg"
server_key: "06fcabdbb520b0f4e0f596d47d04eb3f"
server_key_iv: "c586ff48e985ba0d6d57232c3dd4a0e3"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:21.046808 26104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:21.075057 26104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:21.078897 26104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:21.079146 26104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:21.080073 26104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:21.081542 26104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:21.081655 26104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:21.081753 26104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:21.081825 26104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:21.137271 26104 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:41963
I20260928 03:01:21.137352 26244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:41963 every 8 connection(s)
I20260928 03:01:21.139098 26104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:01:21.139676 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 26104
I20260928 03:01:21.140463 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.182.84:33267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:40561
--local_ip_for_outbound_sockets=127.12.182.67
--tserver_master_addrs=127.12.182.126:46449
--webserver_port=44951
--webserver_interface=127.12.182.67
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.182.84:33267/kms with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:01:21 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:21.218806 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:21.159308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46449 (local address 127.12.182.66:55927)
0928 03:01:21.180896 (+ 21588us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:21.180971 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:21.183913 (+ 2942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:21.194634 (+ 10721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:21.194647 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:21.195380 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:21.196220 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:21.196246 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:21.198595 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:21.198601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:21.200519 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:21.200535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:21.201811 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:21.201819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:21.202085 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:21.206477 (+ 4392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:21.206509 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:21.210462 (+ 3953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:21.214441 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:21.214449 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:21.214464 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:21.215042 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:21.215486 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:21.215491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:21.215495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:21.215730 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:21.216426 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:21.216446 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:21.216796 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 03:01:21.217069 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 03:01:21.217574 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21213,"thread_start_us":25216,"threads_started":1}
I20260928 03:01:21.219749 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:21.159950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:55927 (local address 127.12.182.126:46449)
0928 03:01:21.180133 (+ 20183us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:21.180140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:21.181372 (+ 1232us) server_negotiation.cc:407] Connection header received
0928 03:01:21.193792 (+ 12420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:21.193800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:21.193914 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:21.194027 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:21.197225 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:21.198386 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:21.200707 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:21.201346 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:21.201480 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:21.210735 (+ 9255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:21.210742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:21.210747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:21.210788 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:21.214267 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:21.215185 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:21.215191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:21.215195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:21.215346 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:21.215859 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:21.215864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:21.215868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:21.216219 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:21.216353 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:21.219137 (+ 2784us) server_negotiation.cc:299] Negotiation successful
0928 03:01:21.219364 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20019,"thread_start_us":20195,"threads_started":1}
I20260928 03:01:21.221464 26245 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46449
I20260928 03:01:21.221782 26245 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:21.222589 26245 heartbeater.cc:511] Master 127.12.182.126:46449 requested a full tablet report, sending...
I20260928 03:01:21.224251 25903 ts_manager.cc:194] Registered new tserver with Master: f55efc7b9ac84cca930b34e5f4340d12 (127.12.182.66:41963)
I20260928 03:01:21.225782 25903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:55927
Sep 28 03:01:21 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564481, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:01:21.554942 26100 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:21.555378 26100 heartbeater.cc:511] Master 127.12.182.126:46449 requested a full tablet report, sending...
I20260928 03:01:21.556560 25903 ts_manager.cc:194] Registered new tserver with Master: 3718c60dd92b48bcbf78d9b9e5f0d412 (127.12.182.65:37655)
I20260928 03:01:21.557819 25903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:34011
W20260928 03:01:21.590161 26246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:21.590715 26246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:21.590922 26246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:21.602093 26246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:21.602279 26246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:21.602522 26246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:01:21.617969 26246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.182.84:33267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:40561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=44951
--tserver_master_addrs=127.12.182.126:46449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:21.619742 26246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:21.621416 26246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:21.637106 26254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:21.638676 26255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:21.643592 26246 server_base.cc:1034] running on GCE node
W20260928 03:01:21.644261 26257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:21.645030 26246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:21.656592 26246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:21.659969 26246 hybrid_clock.cc:648] HybridClock initialized: now 1790564481659833 us; error 139 us; skew 500 ppm
I20260928 03:01:21.660409 26246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:21 dist-test-slave-k5rg krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564481, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:21.664595 26246 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:01:21.714030 26246 webserver.cc:460] Webserver started at http://127.12.182.67:44951/ using document root <none> and password file <none>
I20260928 03:01:21.715087 26246 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:21.715301 26246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 03:01:21 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564481, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:01:21.763247 26246 fs_manager.cc:709] Time spent opening directory manager: real 0.023s user 0.012s sys 0.000s
I20260928 03:01:21.774240 26265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:21.777199 26246 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.007s
I20260928 03:01:21.777470 26246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "99f06afcebcd400eb9cb6c9b7704c0f0"
format_stamp: "Formatted at 2026-09-28 03:01:18 on dist-test-slave-k5rg"
server_key: "be99b2a2b7756c9db3881294d916e849"
server_key_iv: "2e6b63faee62cb342604043258022032"
server_key_version: "kuduclusterkey@0"
I20260928 03:01:21.778981 26246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:21.856248 26246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 28 03:01:21 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564481, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for HTTP/127.12.182.84@KRBTEST.COM
I20260928 03:01:21.864558 26246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:21.864936 26246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:21.867390 26246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:21.876384 26246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:21.876561 26246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 03:01:21.876713 26246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:21.876827 26246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:22.101128 26246 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:40561
I20260928 03:01:22.101204 26378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:40561 every 8 connection(s)
I20260928 03:01:22.103164 26246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:01:22.106575 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 26246
Sep 28 03:01:22 dist-test-slave-k5rg krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564481, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:22.143426 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.107602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.12.182.126:46449)
0928 03:01:22.125474 (+ 17872us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.125480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.125500 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:01:22.125564 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.125569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.125709 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.127864 (+ 2155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:22.128028 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.129188 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.131209 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.131903 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.133579 (+ 1676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.135454 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:22.135461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:22.135466 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:22.135508 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:22.139757 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.140797 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.140803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.140806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.141226 (+ 420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.141660 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.141665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.141669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.142019 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:22.142302 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.142830 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.143056 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":17745,"thread_start_us":21007,"threads_started":1}
I20260928 03:01:22.145742 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.120018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:36041 (local address 127.12.182.126:46449)
0928 03:01:22.128303 (+ 8285us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.128309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.128329 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:01:22.128379 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.128383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.128501 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.130584 (+ 2083us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:22.130791 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.132051 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.134458 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.135112 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.135972 (+ 860us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.139283 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:22.139290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:22.139294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:22.139333 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:22.142688 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.143515 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.143520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.143523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.143654 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.144155 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.144160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.144163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.144410 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:22.144577 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.145288 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.145430 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":8163,"thread_start_us":5467,"threads_started":1}
I20260928 03:01:22.146664 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.120701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46449 (local address 127.12.182.67:36041)
0928 03:01:22.121532 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.121579 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:22.123866 (+ 2287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:22.129002 (+ 5136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:22.129014 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:22.129633 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:22.130403 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.130432 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.132224 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.132229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.134266 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.134295 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.135345 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.135352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.135636 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.136307 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:22.136336 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:22.139016 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:22.142825 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:22.142833 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:22.142848 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:22.143379 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:22.143786 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:22.143792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:22.143796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:22.144057 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:22.144532 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:22.144539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:22.144887 (+ 348us) client_negotiation.cc:769] Sending connection context
0928 03:01:22.145180 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 03:01:22.145700 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":340,"threads_started":1}
I20260928 03:01:22.147845 26379 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46449
I20260928 03:01:22.148123 26379 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:22.148758 26379 heartbeater.cc:511] Master 127.12.182.126:46449 requested a full tablet report, sending...
I20260928 03:01:22.150039 25903 ts_manager.cc:194] Registered new tserver with Master: 99f06afcebcd400eb9cb6c9b7704c0f0 (127.12.182.67:40561)
I20260928 03:01:22.150763 25903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:36041
I20260928 03:01:22.158058 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:01:22.172086 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.160732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.12.182.126:46449)
0928 03:01:22.160881 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.160885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.161315 (+ 430us) server_negotiation.cc:407] Connection header received
0928 03:01:22.161456 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.161459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.161556 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.161679 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:22.163080 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.163735 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.164940 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.165324 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.165431 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.167726 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:22.167730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:22.167734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:22.167761 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:22.170061 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.170594 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.170598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.170600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.170697 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.170964 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.170967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.170969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.171207 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:22.171365 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.171656 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.171818 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 03:01:22.176436 25903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:01:22.183651 25903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:01:22.206911 26402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.196609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.12.182.66:41963)
0928 03:01:22.198709 (+ 2100us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.198716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.198742 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:01:22.198862 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.198870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.199099 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.200029 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:22.200272 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.201809 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.204294 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.205774 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.205909 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.206124 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.206406 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.206572 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":2457,"threads_started":1}
I20260928 03:01:22.206962 26401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.196274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:41963 (local address 127.0.0.1:47796)
0928 03:01:22.197354 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.197409 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:22.197586 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:22.199324 (+ 1738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:22.199331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:22.199362 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:22.199811 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.199824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.201949 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.201955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.204137 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.204152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.205971 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.205977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.206166 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.206218 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 03:01:22.206357 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 03:01:22.206533 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":35,"thread_start_us":670,"threads_started":1}
I20260928 03:01:22.207846 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.196678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:40561 (local address 127.0.0.1:42664)
0928 03:01:22.198170 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.198187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:22.198285 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:22.200316 (+ 2031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:22.200320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:22.200347 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:22.200664 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.200675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.202954 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.202959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.205105 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.205115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.206796 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.206803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.206981 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.207011 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 03:01:22.207076 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 03:01:22.207188 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"mutex_wait_us":111,"thread_start_us":1228,"threads_started":1}
I20260928 03:01:22.207945 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.198266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:37655 (local address 127.0.0.1:39322)
0928 03:01:22.200002 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.200021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:22.200134 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:22.200970 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:22.200973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:22.200995 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:22.201280 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.201291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.202809 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.202813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.205284 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.205297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.207215 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.207220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.207332 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.207354 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 03:01:22.207436 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 03:01:22.207552 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1621,"mutex_wait_us":225,"thread_start_us":1471,"threads_started":1}
I20260928 03:01:22.208586 26404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.197338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.12.182.67:40561)
0928 03:01:22.199694 (+ 2356us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.199703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.199733 (+ 30us) server_negotiation.cc:407] Connection header received
0928 03:01:22.199878 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.199886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.200102 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.200834 (+ 732us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:22.201304 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.202774 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.205232 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.206557 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.207297 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.208157 (+ 860us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.208221 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.208344 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2196,"thread_start_us":2046,"threads_started":1}
I20260928 03:01:22.208999 26405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.198047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.12.182.65:37655)
0928 03:01:22.200582 (+ 2535us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.200587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.200606 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:01:22.200674 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.200679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.200816 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.200963 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:22.201477 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.202689 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.205460 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.206756 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.206867 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.207073 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.207494 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.207644 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2408,"thread_start_us":2281,"threads_started":1}
I20260928 03:01:22.211766 26035 tablet_service.cc:1467] Processing CreateTablet for tablet ab9817383140469c9b432f7ec6b0d29d (DEFAULT_TABLE table=test-table [id=eafe5682a1d84fe8bd279118609eaf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:22.211547 26180 tablet_service.cc:1467] Processing CreateTablet for tablet ab9817383140469c9b432f7ec6b0d29d (DEFAULT_TABLE table=test-table [id=eafe5682a1d84fe8bd279118609eaf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:22.211869 26314 tablet_service.cc:1467] Processing CreateTablet for tablet ab9817383140469c9b432f7ec6b0d29d (DEFAULT_TABLE table=test-table [id=eafe5682a1d84fe8bd279118609eaf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:22.212868 26035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9817383140469c9b432f7ec6b0d29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:22.212980 26314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9817383140469c9b432f7ec6b0d29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:22.212990 26180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9817383140469c9b432f7ec6b0d29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:22.223843 26407 tablet_bootstrap.cc:492] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412: Bootstrap starting.
I20260928 03:01:22.227157 26408 tablet_bootstrap.cc:492] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: Bootstrap starting.
I20260928 03:01:22.227794 26407 tablet_bootstrap.cc:654] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:22.229089 26245 heartbeater.cc:503] Master 127.12.182.126:46449 was elected leader, sending a full tablet report...
I20260928 03:01:22.229355 26407 log.cc:824] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:22.230602 26408 tablet_bootstrap.cc:654] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:22.232333 26409 tablet_bootstrap.cc:492] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: Bootstrap starting.
I20260928 03:01:22.234093 26408 log.cc:824] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:22.235720 26409 tablet_bootstrap.cc:654] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:22.236862 26409 log.cc:824] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:22.237149 26407 tablet_bootstrap.cc:492] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412: No bootstrap required, opened a new log
I20260928 03:01:22.237174 26408 tablet_bootstrap.cc:492] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: No bootstrap required, opened a new log
I20260928 03:01:22.237439 26407 ts_tablet_manager.cc:1397] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260928 03:01:22.237437 26408 ts_tablet_manager.cc:1397] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 03:01:22.239795 26409 tablet_bootstrap.cc:492] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: No bootstrap required, opened a new log
I20260928 03:01:22.240029 26409 ts_tablet_manager.cc:1397] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 03:01:22.244555 26407 raft_consensus.cc:348] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55efc7b9ac84cca930b34e5f4340d12" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41963 } } peers { permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 } } peers { permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 } }
I20260928 03:01:22.244943 26407 raft_consensus.cc:374] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:22.245038 26407 raft_consensus.cc:729] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3718c60dd92b48bcbf78d9b9e5f0d412, State: Initialized, Role: FOLLOWER
I20260928 03:01:22.245586 26407 consensus_queue.cc:260] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55efc7b9ac84cca930b34e5f4340d12" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41963 } } peers { permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 } } peers { permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 } }
I20260928 03:01:22.247152 26408 raft_consensus.cc:348] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55efc7b9ac84cca930b34e5f4340d12" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41963 } } peers { permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 } } peers { permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 } }
I20260928 03:01:22.247550 26408 raft_consensus.cc:374] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:22.247630 26408 raft_consensus.cc:729] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55efc7b9ac84cca930b34e5f4340d12, State: Initialized, Role: FOLLOWER
I20260928 03:01:22.248183 26408 consensus_queue.cc:260] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55efc7b9ac84cca930b34e5f4340d12" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41963 } } peers { permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 } } peers { permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 } }
I20260928 03:01:22.249924 26408 ts_tablet_manager.cc:1428] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260928 03:01:22.249713 26409 raft_consensus.cc:348] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55efc7b9ac84cca930b34e5f4340d12" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41963 } } peers { permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 } } peers { permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 } }
I20260928 03:01:22.250100 26409 raft_consensus.cc:374] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:22.250190 26409 raft_consensus.cc:729] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f06afcebcd400eb9cb6c9b7704c0f0, State: Initialized, Role: FOLLOWER
I20260928 03:01:22.250828 26100 heartbeater.cc:503] Master 127.12.182.126:46449 was elected leader, sending a full tablet report...
I20260928 03:01:22.250808 26409 consensus_queue.cc:260] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55efc7b9ac84cca930b34e5f4340d12" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41963 } } peers { permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 } } peers { permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 } }
I20260928 03:01:22.251477 26407 ts_tablet_manager.cc:1428] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260928 03:01:22.252863 26379 heartbeater.cc:503] Master 127.12.182.126:46449 was elected leader, sending a full tablet report...
I20260928 03:01:22.253407 26409 ts_tablet_manager.cc:1428] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260928 03:01:22.261373 26101 tablet.cc:2369] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:01:22.266732 26414 raft_consensus.cc:482] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:01:22.266963 26414 raft_consensus.cc:504] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55efc7b9ac84cca930b34e5f4340d12" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41963 } } peers { permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 } } peers { permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 } }
I20260928 03:01:22.268518 26414 leader_election.cc:288] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f06afcebcd400eb9cb6c9b7704c0f0 (127.12.182.67:40561), 3718c60dd92b48bcbf78d9b9e5f0d412 (127.12.182.65:37655)
I20260928 03:01:22.274767 26404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.268812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:45785 (local address 127.12.182.67:40561)
0928 03:01:22.269056 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.269062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.269896 (+ 834us) server_negotiation.cc:407] Connection header received
0928 03:01:22.270060 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.270064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.270118 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.270187 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:22.270742 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.271409 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.273116 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.273886 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.274022 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.274091 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.274424 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.274542 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260928 03:01:22.274775 26405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.269008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:50927 (local address 127.12.182.65:37655)
0928 03:01:22.269213 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.269220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.269808 (+ 588us) server_negotiation.cc:407] Connection header received
0928 03:01:22.269998 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.270001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.270056 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.270127 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:22.270687 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.271372 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.273088 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.273848 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.273987 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.274057 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.274445 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.274560 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:01:22.274899 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.268848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:40561 (local address 127.12.182.66:45785)
0928 03:01:22.269798 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.269812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:22.269936 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:22.270208 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:22.270212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:22.270239 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:22.270595 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.270604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.271535 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.271543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.272984 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.272995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.274089 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.274095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.274282 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.274305 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 03:01:22.274387 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 03:01:22.274503 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":58,"thread_start_us":367,"threads_started":1}
I20260928 03:01:22.275305 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.268810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:37655 (local address 127.12.182.66:50927)
0928 03:01:22.269715 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.269731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:22.269867 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:22.270149 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:22.270154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:22.270176 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:22.270561 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.270574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.271492 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.271497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.272956 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:22.272968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.274049 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.274056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:22.274297 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.274325 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 03:01:22.274400 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 03:01:22.274513 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":63,"thread_start_us":703,"threads_started":1}
I20260928 03:01:22.275786 26334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9817383140469c9b432f7ec6b0d29d" candidate_uuid: "f55efc7b9ac84cca930b34e5f4340d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" is_pre_election: true
I20260928 03:01:22.276157 26055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9817383140469c9b432f7ec6b0d29d" candidate_uuid: "f55efc7b9ac84cca930b34e5f4340d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" is_pre_election: true
I20260928 03:01:22.276268 26334 raft_consensus.cc:2454] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f55efc7b9ac84cca930b34e5f4340d12 in term 0.
I20260928 03:01:22.276551 26055 raft_consensus.cc:2454] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f55efc7b9ac84cca930b34e5f4340d12 in term 0.
I20260928 03:01:22.276832 26134 leader_election.cc:302] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f06afcebcd400eb9cb6c9b7704c0f0, f55efc7b9ac84cca930b34e5f4340d12; no voters:
I20260928 03:01:22.277140 26414 raft_consensus.cc:2781] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:01:22.277251 26414 raft_consensus.cc:482] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:01:22.277339 26414 raft_consensus.cc:3037] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:22.279433 26414 raft_consensus.cc:504] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55efc7b9ac84cca930b34e5f4340d12" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41963 } } peers { permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 } } peers { permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 } }
I20260928 03:01:22.279954 26414 leader_election.cc:288] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [CANDIDATE]: Term 1 election: Requested vote from peers 99f06afcebcd400eb9cb6c9b7704c0f0 (127.12.182.67:40561), 3718c60dd92b48bcbf78d9b9e5f0d412 (127.12.182.65:37655)
I20260928 03:01:22.280309 26334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9817383140469c9b432f7ec6b0d29d" candidate_uuid: "f55efc7b9ac84cca930b34e5f4340d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0"
I20260928 03:01:22.280378 26055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9817383140469c9b432f7ec6b0d29d" candidate_uuid: "f55efc7b9ac84cca930b34e5f4340d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412"
I20260928 03:01:22.280462 26334 raft_consensus.cc:3037] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:22.280583 26055 raft_consensus.cc:3037] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:22.282547 26334 raft_consensus.cc:2454] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f55efc7b9ac84cca930b34e5f4340d12 in term 1.
I20260928 03:01:22.282775 26055 raft_consensus.cc:2454] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f55efc7b9ac84cca930b34e5f4340d12 in term 1.
I20260928 03:01:22.283224 26133 leader_election.cc:302] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3718c60dd92b48bcbf78d9b9e5f0d412, f55efc7b9ac84cca930b34e5f4340d12; no voters:
I20260928 03:01:22.283612 26414 raft_consensus.cc:2781] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:01:22.284024 26414 raft_consensus.cc:686] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 1 LEADER]: Becoming Leader. State: Replica: f55efc7b9ac84cca930b34e5f4340d12, State: Running, Role: LEADER
I20260928 03:01:22.284456 26414 consensus_queue.cc:237] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55efc7b9ac84cca930b34e5f4340d12" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41963 } } peers { permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 } } peers { permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 } }
I20260928 03:01:22.288870 25903 catalog_manager.cc:5495] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 reported cstate change: term changed from 0 to 1, leader changed from <none> to f55efc7b9ac84cca930b34e5f4340d12 (127.12.182.66). New cstate: current_term: 1 leader_uuid: "f55efc7b9ac84cca930b34e5f4340d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55efc7b9ac84cca930b34e5f4340d12" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 41963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 } health_report { overall_health: UNKNOWN } } }
I20260928 03:01:22.306598 26402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.301451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.12.182.66:41963)
0928 03:01:22.301656 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.301661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.301734 (+ 73us) server_negotiation.cc:407] Connection header received
0928 03:01:22.301942 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.301946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.302025 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.302140 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:01:22.302527 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.303250 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.304622 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.305000 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.305100 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.305556 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:01:22.306055 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:01:22.306181 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.306287 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.306360 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 03:01:22.313911 26334 raft_consensus.cc:1261] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0 [term 1 FOLLOWER]: Refusing update from remote peer f55efc7b9ac84cca930b34e5f4340d12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:01:22.313937 26055 raft_consensus.cc:1261] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412 [term 1 FOLLOWER]: Refusing update from remote peer f55efc7b9ac84cca930b34e5f4340d12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:01:22.314806 26418 consensus_queue.cc:1035] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f06afcebcd400eb9cb6c9b7704c0f0" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:01:22.315110 26414 consensus_queue.cc:1035] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3718c60dd92b48bcbf78d9b9e5f0d412" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:01:22.324884 26421 mvcc.cc:205] Tried to move back new op lower bound from 7334152119548387328 to 7334152119440642048. Current Snapshot: MvccSnapshot[applied={T|T < 7334152119548387328}]
I20260928 03:01:22.328207 26422 mvcc.cc:205] Tried to move back new op lower bound from 7334152119548387328 to 7334152119440642048. Current Snapshot: MvccSnapshot[applied={T|T < 7334152119548387328}]
I20260928 03:01:22.330775 26423 mvcc.cc:205] Tried to move back new op lower bound from 7334152119548387328 to 7334152119440642048. Current Snapshot: MvccSnapshot[applied={T|T < 7334152119548387328}]
I20260928 03:01:22.337464 25903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:01:22.337699 25903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:01:22.340829 25903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3ea60c0e12c4e118bec23ccc7321aa5: Sending DeleteTablet for 3 replicas of tablet ab9817383140469c9b432f7ec6b0d29d
I20260928 03:01:22.341979 26035 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9817383140469c9b432f7ec6b0d29d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:22 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:39322
I20260928 03:01:22.341969 26314 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9817383140469c9b432f7ec6b0d29d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:22 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:42664
I20260928 03:01:22.342778 26180 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9817383140469c9b432f7ec6b0d29d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:22 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:47796
I20260928 03:01:22.346458 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 25940
I20260928 03:01:22.353749 26432 tablet_replica.cc:330] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: stopping tablet replica
I20260928 03:01:22.354203 26432 raft_consensus.cc:2229] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:01:22.356097 26432 raft_consensus.cc:2258] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:22.357991 26432 ts_tablet_manager.cc:1905] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:01:22.360350 26431 tablet_replica.cc:330] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: stopping tablet replica
I20260928 03:01:22.360965 26430 tablet_replica.cc:330] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412: stopping tablet replica
I20260928 03:01:22.360994 26431 raft_consensus.cc:2229] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:01:22.361337 26430 raft_consensus.cc:2229] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:01:22.361517 26431 raft_consensus.cc:2258] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:22.361804 26430 raft_consensus.cc:2258] T ab9817383140469c9b432f7ec6b0d29d P 3718c60dd92b48bcbf78d9b9e5f0d412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:22.363608 26431 ts_tablet_manager.cc:1905] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:01:22.365553 26405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.347741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.12.182.65:37655)
0928 03:01:22.347917 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.347923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.349008 (+ 1085us) server_negotiation.cc:407] Connection header received
0928 03:01:22.349200 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.349205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.349284 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.349383 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:22.351478 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.352612 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.354492 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.355215 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.355343 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.358778 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:22.358794 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:22.358813 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:22.358856 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:22.362785 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.363420 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.363429 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.363432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.363589 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.364009 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.364014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.364017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.364430 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:22.364707 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.364990 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.365196 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:01:22.366299 26095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:22.370379 26432 ts_tablet_manager.cc:1918] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:01:22.370546 26432 log.cc:1196] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/ab9817383140469c9b432f7ec6b0d29d
I20260928 03:01:22.371071 26432 ts_tablet_manager.cc:1939] T ab9817383140469c9b432f7ec6b0d29d P f55efc7b9ac84cca930b34e5f4340d12: Deleting consensus metadata
I20260928 03:01:22.372831 25888 catalog_manager.cc:4841] TS f55efc7b9ac84cca930b34e5f4340d12 (127.12.182.66:41963): tablet ab9817383140469c9b432f7ec6b0d29d (table test-table [id=eafe5682a1d84fe8bd279118609eaf14]) successfully deleted
I20260928 03:01:22.374536 26431 ts_tablet_manager.cc:1918] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:01:22.374737 26431 log.cc:1196] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/ab9817383140469c9b432f7ec6b0d29d
I20260928 03:01:22.375236 26431 ts_tablet_manager.cc:1939] T ab9817383140469c9b432f7ec6b0d29d P 99f06afcebcd400eb9cb6c9b7704c0f0: Deleting consensus metadata
I20260928 03:01:22.376642 25889 catalog_manager.cc:4841] TS 99f06afcebcd400eb9cb6c9b7704c0f0 (127.12.182.67:40561): tablet ab9817383140469c9b432f7ec6b0d29d (table test-table [id=eafe5682a1d84fe8bd279118609eaf14]) successfully deleted
I20260928 03:01:22.466116 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 25940
W20260928 03:01:22.479249 25888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.182.65:37655 (error 108)
I20260928 03:01:22.479866 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 26104
I20260928 03:01:22.481177 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.480599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:37655 (local address 127.0.0.1:39344)
0928 03:01:22.480884 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.481009 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
W20260928 03:01:22.481439 25888 catalog_manager.cc:4584] TS 3718c60dd92b48bcbf78d9b9e5f0d412 (127.12.182.65:37655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab9817383140469c9b432f7ec6b0d29d: Network error: Client connection negotiation failed: client connection to 127.12.182.65:37655: connect: Connection refused (error 111)
I20260928 03:01:22.499969 26402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.480599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.12.182.66:41963)
0928 03:01:22.480773 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.480780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.480917 (+ 137us) server_negotiation.cc:407] Connection header received
0928 03:01:22.481119 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.481125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.481214 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.481337 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:22.483082 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.483990 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.485842 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.486260 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.486364 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.494429 (+ 8065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:22.494443 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:22.494460 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:22.494499 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:22.497286 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.498087 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.498094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.498096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.498254 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.498606 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.498609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.498611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.498955 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:22.499161 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.499475 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.499654 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:01:22.500664 26240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:22.534965 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.534483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:37655 (local address 127.0.0.1:39350)
0928 03:01:22.534709 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.534817 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 03:01:22.535948 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.535623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:37655 (local address 127.0.0.1:39352)
0928 03:01:22.535746 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.535831 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 03:01:22.596563 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.596072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:37655 (local address 127.0.0.1:39364)
0928 03:01:22.596306 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.596403 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 03:01:22.597520 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.597211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:37655 (local address 127.0.0.1:39378)
0928 03:01:22.597340 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.597412 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 03:01:22.611449 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 26104
I20260928 03:01:22.625528 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 26246
I20260928 03:01:22.637956 26404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.626265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.12.182.67:40561)
0928 03:01:22.626430 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:22.626434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:22.626451 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:01:22.626578 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:22.626582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:22.626640 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:22.626717 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:22.627908 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.628613 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.629942 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:22.630318 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:22.630407 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:22.632782 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:22.632796 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:22.632811 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:22.632855 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:22.635482 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.636131 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.636138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.636140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.636263 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:22.636576 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:22.636580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:22.636583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:22.636965 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:22.637183 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:22.637426 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 03:01:22.637573 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:01:22.638620 26374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:22.677824 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.677347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:37655 (local address 127.0.0.1:32850)
0928 03:01:22.677548 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.677664 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 03:01:22.678812 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.678434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:37655 (local address 127.0.0.1:32856)
0928 03:01:22.678626 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.678704 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 03:01:22.723937 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 26246
I20260928 03:01:22.738237 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 25870
I20260928 03:01:22.739537 25933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:22.852577 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:22.852119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:37655 (local address 127.0.0.1:32860)
0928 03:01:22.852257 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:22.852353 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 03:01:22.853101 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 25870
I20260928 03:01:22.873420 13017 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 03:01:23.207948 13017 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 03:01:23.208144 13017 mini_ranger.cc:66] Stopping Ranger...
I20260928 03:01:23.542533 13017 mini_ranger.cc:68] Stopped Ranger
2026-09-28 03:01:23.542 UTC [24922] LOG: received smart shutdown request
2026-09-28 03:01:23.545 UTC [24922] LOG: background worker "logical replication launcher" (PID 24931) exited with exit code 1
2026-09-28 03:01:23.545 UTC [24926] LOG: shutting down
2026-09-28 03:01:23.694 UTC [24922] LOG: database system is shut down
2026-09-28T03:01:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50300 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:01:23 dist-test-slave-k5rg krb5kdc[26446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:01:23 dist-test-slave-k5rg krb5kdc[26446](info): set up 2 sockets
Sep 28 03:01:23 dist-test-slave-k5rg krb5kdc[26446](info): commencing operation
krb5kdc: starting...
W20260928 03:01:24.849788 13017 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:01:24 dist-test-slave-k5rg krb5kdc[26446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:01:25.049345 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:35651
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:35651,127.12.182.125:43307,127.12.182.124:46113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:25.358001 26459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:25.358274 26459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:25.358407 26459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:25.365350 26459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:01:25.365453 26459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:25.365531 26459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:25.365623 26459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:01:25.365687 26459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:01:25.375221 26459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:35651,127.12.182.125:43307,127.12.182.124:46113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:35651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:25.376467 26459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:25.377732 26459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:25.388814 26464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:25.388855 26465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:25.388878 26467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:25.389904 26459 server_base.cc:1034] running on GCE node
I20260928 03:01:25.390599 26459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:25.391835 26459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:25.393038 26459 hybrid_clock.cc:648] HybridClock initialized: now 1790564485392989 us; error 41 us; skew 500 ppm
I20260928 03:01:25.393368 26459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:25 dist-test-slave-k5rg krb5kdc[26446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:25.396236 26459 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:01:25.397624 26459 webserver.cc:460] Webserver started at http://127.12.182.126:41561/ using document root <none> and password file <none>
I20260928 03:01:25.398237 26459 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:25.398337 26459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:25.398567 26459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:25.401307 26459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "06afd1cf7ce244459c16e05f0cdd2f51"
format_stamp: "Formatted at 2026-09-28 03:01:25 on dist-test-slave-k5rg"
server_key: "770f3f04e88ad5d97e5cb28a49dfe93e"
server_key_iv: "b8c2b94f1559141aa83585d7541bc21e"
server_key_version: "encryptionkey@0"
I20260928 03:01:25.402046 26459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "06afd1cf7ce244459c16e05f0cdd2f51"
format_stamp: "Formatted at 2026-09-28 03:01:25 on dist-test-slave-k5rg"
server_key: "770f3f04e88ad5d97e5cb28a49dfe93e"
server_key_iv: "b8c2b94f1559141aa83585d7541bc21e"
server_key_version: "encryptionkey@0"
I20260928 03:01:25.406699 26459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:01:25.409762 26474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:25.410992 26459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:01:25.411144 26459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "06afd1cf7ce244459c16e05f0cdd2f51"
format_stamp: "Formatted at 2026-09-28 03:01:25 on dist-test-slave-k5rg"
server_key: "770f3f04e88ad5d97e5cb28a49dfe93e"
server_key_iv: "b8c2b94f1559141aa83585d7541bc21e"
server_key_version: "encryptionkey@0"
I20260928 03:01:25.411330 26459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:25.440693 26459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:25.444665 26459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:25.444929 26459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:25.461843 26459 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:35651
I20260928 03:01:25.461877 26525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:35651 every 8 connection(s)
I20260928 03:01:25.463093 26459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:01:25.466569 26526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:25.467338 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 26459
I20260928 03:01:25.467540 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:01:25.468135 13017 external_mini_cluster.cc:1428] Setting key 5d25152ec2a0fff3547698a063f5c314
I20260928 03:01:25.472770 26526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.125@KRBTEST.COM" created.
Sep 28 03:01:25 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:25.496104 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.475584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:35651 (local address 127.0.0.1:54482)
0928 03:01:25.476316 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.476370 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:25.478325 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:25.479157 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:25.479162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:25.479683 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:25.479986 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:25.480002 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.482271 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.482275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:25.483351 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:25.483363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.484574 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.484580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:25.484711 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:25.485335 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:25.485356 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:25.489693 (+ 4337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:25.492645 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:25.492655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:25.492668 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:25.493037 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:25.493421 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:25.493426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:25.493430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:25.493578 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:25.494355 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:25.494362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:25.494538 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 03:01:25.494740 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 03:01:25.495125 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":296,"threads_started":1}
I20260928 03:01:25.496134 26530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.476752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.12.182.126:35651)
0928 03:01:25.477382 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:25.477393 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:25.477439 (+ 46us) server_negotiation.cc:407] Connection header received
0928 03:01:25.478664 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:25.478671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:25.478937 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:25.479122 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:25.480171 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.481952 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.483494 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.484360 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.484563 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:25.489913 (+ 5350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:25.489921 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:25.489933 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:25.489962 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:25.492513 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:25.493154 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:25.493161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:25.493164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:25.493304 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:25.493682 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:25.493686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:25.493689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:25.494210 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:25.494414 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:25.494926 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 03:01:25.495302 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":442,"thread_start_us":294,"threads_started":1}
I20260928 03:01:25.499071 26526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:01:25.500098 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.499761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43122)
0928 03:01:25.499909 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.500007 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 03:01:25.500363 26476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
I20260928 03:01:25.501824 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.501473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43136)
0928 03:01:25.501629 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.501708 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 03:01:25.501997 26526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:43307: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab.
Entry for principal kudu/127.12.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab.
I20260928 03:01:25.513432 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.125:43307
--webserver_interface=127.12.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab
--principal=kudu/127.12.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:35651,127.12.182.125:43307,127.12.182.124:46113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:01:25.551288 26526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } attempt: 1
I20260928 03:01:25.552461 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.552013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43140)
0928 03:01:25.552217 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.552323 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 03:01:25.553354 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.553022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43156)
0928 03:01:25.553157 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.553226 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 03:01:25.553519 26526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:43307: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
I20260928 03:01:25.621758 26526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } attempt: 2
I20260928 03:01:25.622922 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.622443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43166)
0928 03:01:25.622664 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.622754 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 03:01:25.623836 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.623471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43176)
0928 03:01:25.623608 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.623691 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 03:01:25.624001 26526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:43307: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
I20260928 03:01:25.715243 26526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } attempt: 3
I20260928 03:01:25.716356 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.715917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43180)
0928 03:01:25.716123 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.716210 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 03:01:25.717317 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.716951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43192)
0928 03:01:25.717099 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.717178 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 03:01:25.717484 26526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:43307: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
W20260928 03:01:25.817237 26533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:25.817507 26533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:25.817655 26533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:25.824671 26533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:01:25.824779 26533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:25.824862 26533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:25.824947 26533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:01:25.825006 26533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:01:25.834599 26533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:35651,127.12.182.125:43307,127.12.182.124:46113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab
--principal=kudu/127.12.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.125:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:25.835984 26533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:25.837177 26533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:25.847738 26541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:25.847826 26538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:25.847756 26539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:25.848512 26533 server_base.cc:1034] running on GCE node
I20260928 03:01:25.849210 26533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:25.850469 26533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:25.851656 26533 hybrid_clock.cc:648] HybridClock initialized: now 1790564485851610 us; error 40 us; skew 500 ppm
I20260928 03:01:25.851959 26533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:25 dist-test-slave-k5rg krb5kdc[26446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:25.854746 26533 init.cc:377] Logged in from keytab as kudu/127.12.182.125@KRBTEST.COM (short username kudu)
I20260928 03:01:25.856037 26533 webserver.cc:460] Webserver started at http://127.12.182.125:36177/ using document root <none> and password file <none>
I20260928 03:01:25.856606 26533 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:25.856693 26533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:25.856930 26533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:25.859612 26533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/data/instance:
uuid: "1e87781cb5424fad937006a16697cc27"
format_stamp: "Formatted at 2026-09-28 03:01:25 on dist-test-slave-k5rg"
server_key: "832146e0989acea96c3798bd5f6c2b37"
server_key_iv: "2112c330e7da5d00d333aab5d8834d28"
server_key_version: "encryptionkey@0"
I20260928 03:01:25.860298 26533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/wal/instance:
uuid: "1e87781cb5424fad937006a16697cc27"
format_stamp: "Formatted at 2026-09-28 03:01:25 on dist-test-slave-k5rg"
server_key: "832146e0989acea96c3798bd5f6c2b37"
server_key_iv: "2112c330e7da5d00d333aab5d8834d28"
server_key_version: "encryptionkey@0"
I20260928 03:01:25.860685 26526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } attempt: 4
I20260928 03:01:25.861683 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.861169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43194)
0928 03:01:25.861444 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.861519 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260928 03:01:25.862564 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.862232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43208)
0928 03:01:25.862369 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.862453 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 03:01:25.862773 26526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:43307: Network error: Client connection negotiation failed: client connection to 127.12.182.125:43307: connect: Connection refused (error 111)
I20260928 03:01:25.865005 26533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 03:01:25.868063 26548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:25.869215 26533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 03:01:25.869350 26533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/wal
uuid: "1e87781cb5424fad937006a16697cc27"
format_stamp: "Formatted at 2026-09-28 03:01:25 on dist-test-slave-k5rg"
server_key: "832146e0989acea96c3798bd5f6c2b37"
server_key_iv: "2112c330e7da5d00d333aab5d8834d28"
server_key_version: "encryptionkey@0"
I20260928 03:01:25.869509 26533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:25.891400 26533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:25.895117 26533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:25.895366 26533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:25.911515 26533 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.125:43307
I20260928 03:01:25.911554 26599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.125:43307 every 8 connection(s)
I20260928 03:01:25.912672 26533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
I20260928 03:01:25.915938 26600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:25.920873 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 26533
I20260928 03:01:25.921038 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-1/wal/instance
I20260928 03:01:25.921547 13017 external_mini_cluster.cc:1428] Setting key a90b6ccab2b0e483461db2977546011d
I20260928 03:01:25.922462 26600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:01:25 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.124@KRBTEST.COM; defaulting to no policy
I20260928 03:01:25.943120 26530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.925396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.12.182.126:35651)
0928 03:01:25.925595 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:25.925623 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:25.926743 (+ 1120us) server_negotiation.cc:407] Connection header received
0928 03:01:25.928969 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:25.928973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:25.929047 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:25.929130 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:25.930588 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.931594 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.933039 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.933677 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.933791 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:25.937020 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:25.937027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:25.937032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:25.937072 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:25.940283 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:25.941025 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:25.941032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:25.941035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:25.941172 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:25.941572 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:25.941577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:25.941580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:25.941950 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:25.942106 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:25.942703 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 03:01:25.942889 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 03:01:25.943996 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.925838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:35651 (local address 127.0.0.1:54488)
0928 03:01:25.926574 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.926621 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:25.928662 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:25.929373 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:25.929381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:25.929904 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:25.930411 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:25.930429 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.931753 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.931757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:25.932912 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:25.932925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.933871 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.933876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:25.934065 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:25.934669 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:25.934690 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:25.936799 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:25.940427 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:25.940433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:25.940444 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:25.940914 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:25.941300 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:25.941303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:25.941306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:25.941470 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:25.942061 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:25.942067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:25.942404 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 03:01:25.942618 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 03:01:25.942977 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":284,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.124@KRBTEST.COM" created.
I20260928 03:01:25.945844 26600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:01:25 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
I20260928 03:01:25.960038 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.946597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43222)
0928 03:01:25.946773 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.946792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:25.946886 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:25.947931 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:25.947935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:25.948324 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:25.948498 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:25.948508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.949924 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.949929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:25.951109 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:25.951123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.951898 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.951904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:25.952072 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:25.952583 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:25.952599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:25.954730 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:25.957557 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:25.957562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:25.957565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:25.958054 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:25.958386 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:25.958390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:25.958393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:25.958565 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:25.959091 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:25.959096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:25.959234 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 03:01:25.959387 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 03:01:25.959655 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 03:01:25.960053 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.946886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.12.182.125:43307)
0928 03:01:25.947541 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:25.947547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:25.947571 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:01:25.947610 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:25.947616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:25.947786 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:25.947893 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:25.948656 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.949800 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.951238 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:25.951737 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:25.951858 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:25.954930 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:25.954940 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:25.954952 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:25.954983 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:25.957428 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:25.958162 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:25.958167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:25.958170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:25.958276 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:25.958663 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:25.958666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:25.958668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:25.958973 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:25.959125 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:25.959500 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 03:01:25.959753 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":501,"thread_start_us":370,"threads_started":1}
I20260928 03:01:25.962463 26600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:01:25.963402 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.963002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51044)
0928 03:01:25.963156 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.963254 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 03:01:25.963604 26551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
I20260928 03:01:25.964996 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:25.964666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51046)
0928 03:01:25.964784 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:25.964876 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 03:01:25.965193 26600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:46113: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab.
Entry for principal kudu/127.12.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab.
I20260928 03:01:25.972568 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.124:46113
--webserver_interface=127.12.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab
--principal=kudu/127.12.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:35651,127.12.182.125:43307,127.12.182.124:46113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:01:26.014465 26600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } attempt: 1
I20260928 03:01:26.015779 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.015147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51054)
0928 03:01:26.015428 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.015566 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 03:01:26.016929 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.016504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51060)
0928 03:01:26.016681 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.016779 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 03:01:26.017182 26600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:46113: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
I20260928 03:01:26.085422 26600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } attempt: 2
I20260928 03:01:26.086573 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.086059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51066)
0928 03:01:26.086269 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.086387 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 03:01:26.087577 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.087210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51068)
0928 03:01:26.087356 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.087441 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 03:01:26.087809 26600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:46113: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
I20260928 03:01:26.162078 26526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } attempt: 5
Sep 28 03:01:26 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
I20260928 03:01:26.179000 26600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } attempt: 3
I20260928 03:01:26.180117 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.179660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51074)
0928 03:01:26.179845 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.179932 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 03:01:26.181243 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.180739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51076)
0928 03:01:26.180997 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.181085 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260928 03:01:26.181577 26600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:46113: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
I20260928 03:01:26.181790 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.162851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43228)
0928 03:01:26.163088 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.163112 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:26.163257 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:26.163622 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:26.163627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:26.164130 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:26.164376 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.164391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.165471 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.165477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.166917 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.166934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.167687 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.167693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.167869 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.172918 (+ 5049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:26.172942 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:26.175615 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:26.178585 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.178590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.178594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.179568 (+ 974us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.180009 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.180014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.180017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.180187 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.180830 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:26.180835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:26.180984 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 03:01:26.181115 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 03:01:26.181416 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 03:01:26.181811 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.162979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.12.182.125:43307)
0928 03:01:26.163151 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.163156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.163174 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:01:26.163412 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.163416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.163480 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.163553 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.164575 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.165323 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.167063 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.167527 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.167629 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.175880 (+ 8251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.175886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.175889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.175925 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.178439 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.179726 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.179733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.179737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.179888 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.180327 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.180333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.180336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.180701 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.180839 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.181252 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.181468 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 03:01:26.182687 26526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:01:26.183693 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.183289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51088)
0928 03:01:26.183461 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.183546 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 03:01:26.184625 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.184291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51098)
0928 03:01:26.184454 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.184521 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 03:01:26.184842 26526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:46113: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
I20260928 03:01:26.236093 26526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } attempt: 1
I20260928 03:01:26.237191 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.236721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51112)
0928 03:01:26.236947 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.237029 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 03:01:26.238153 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.237840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51122)
0928 03:01:26.237985 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.238056 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 03:01:26.238394 26526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:46113: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
W20260928 03:01:26.279397 26607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:26.279655 26607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:26.279768 26607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:26.286767 26607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:01:26.286863 26607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:26.286931 26607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:26.287006 26607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:01:26.287063 26607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:01:26.296438 26607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:35651,127.12.182.125:43307,127.12.182.124:46113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab
--principal=kudu/127.12.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.124:46113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:26.297653 26607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:26.298847 26607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 03:01:26.306613 26526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } attempt: 2
I20260928 03:01:26.307734 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.307244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51124)
0928 03:01:26.307464 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.307551 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 03:01:26.309059 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.308576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51134)
0928 03:01:26.308764 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.308910 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 03:01:26.309285 26526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:46113: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
W20260928 03:01:26.309685 26613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:26.309932 26615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:26.309952 26612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:26.310988 26607 server_base.cc:1034] running on GCE node
I20260928 03:01:26.311652 26607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:26.312849 26607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:26.314039 26607 hybrid_clock.cc:648] HybridClock initialized: now 1790564486313996 us; error 32 us; skew 500 ppm
I20260928 03:01:26.314361 26607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:26 dist-test-slave-k5rg krb5kdc[26446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564486, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:26.317135 26607 init.cc:377] Logged in from keytab as kudu/127.12.182.124@KRBTEST.COM (short username kudu)
I20260928 03:01:26.318461 26607 webserver.cc:460] Webserver started at http://127.12.182.124:35825/ using document root <none> and password file <none>
I20260928 03:01:26.319049 26607 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:26.319154 26607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:26.319404 26607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:26.322157 26607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/data/instance:
uuid: "4770476f67dd40c3a585fa01c9de5538"
format_stamp: "Formatted at 2026-09-28 03:01:26 on dist-test-slave-k5rg"
server_key: "98d4eb7024897e82cfc8c8d611ce6e98"
server_key_iv: "0eb30ef3b955ece3ce6418c05e1cd4a7"
server_key_version: "encryptionkey@0"
I20260928 03:01:26.322891 26607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/wal/instance:
uuid: "4770476f67dd40c3a585fa01c9de5538"
format_stamp: "Formatted at 2026-09-28 03:01:26 on dist-test-slave-k5rg"
server_key: "98d4eb7024897e82cfc8c8d611ce6e98"
server_key_iv: "0eb30ef3b955ece3ce6418c05e1cd4a7"
server_key_version: "encryptionkey@0"
I20260928 03:01:26.324812 26600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } attempt: 4
I20260928 03:01:26.325843 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.325376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51136)
0928 03:01:26.325573 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.325681 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 03:01:26.326660 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.326366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51148)
0928 03:01:26.326498 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.326563 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 03:01:26.326901 26600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:46113: Network error: Client connection negotiation failed: client connection to 127.12.182.124:46113: connect: Connection refused (error 111)
I20260928 03:01:26.327827 26607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:01:26.330914 26622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:26.332103 26607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260928 03:01:26.332248 26607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/wal
uuid: "4770476f67dd40c3a585fa01c9de5538"
format_stamp: "Formatted at 2026-09-28 03:01:26 on dist-test-slave-k5rg"
server_key: "98d4eb7024897e82cfc8c8d611ce6e98"
server_key_iv: "0eb30ef3b955ece3ce6418c05e1cd4a7"
server_key_version: "encryptionkey@0"
I20260928 03:01:26.332427 26607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:26.364912 26607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:26.368736 26607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:26.368983 26607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:26.385102 26607 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.124:46113
I20260928 03:01:26.385107 26673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.124:46113 every 8 connection(s)
I20260928 03:01:26.386360 26607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
I20260928 03:01:26.389780 26674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:26.390769 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 26607
I20260928 03:01:26.390934 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564295078766-13017-0/minicluster-data/master-2/wal/instance
I20260928 03:01:26.391374 13017 external_mini_cluster.cc:1428] Setting key b2fec15a0ea354a8e5e2e2fc3be444b2
I20260928 03:01:26.399063 26674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:01:26 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
Sep 28 03:01:26 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564486, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:26.414531 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.393129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.12.182.124:46113)
0928 03:01:26.394179 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.394193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.394246 (+ 53us) server_negotiation.cc:407] Connection header received
0928 03:01:26.396029 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.396056 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.396534 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.396971 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.398332 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.400246 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.401519 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.402561 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.402756 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.407041 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.407047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.407056 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.407090 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.410496 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.411271 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.411279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.411286 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.411478 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.411818 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.411823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.411826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.412290 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.412554 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.412974 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.413348 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":439,"threads_started":1}
I20260928 03:01:26.415560 26526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } attempt: 3
I20260928 03:01:26.417575 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.401760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:35651 (local address 127.0.0.1:54498)
0928 03:01:26.402423 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.402458 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:26.402603 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:26.403179 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:26.403186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:26.403926 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:26.404259 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.404271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.405258 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.405264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.406420 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.406432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.407303 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.407309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.407506 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.408156 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:26.408179 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:26.410891 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:26.413859 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.413868 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.413873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.414383 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.415628 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.415633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.415636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.415876 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.416506 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:26.416513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:26.416704 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 03:01:26.416931 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 03:01:26.417215 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":348,"threads_started":1}
I20260928 03:01:26.417997 26530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.401675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.12.182.126:35651)
0928 03:01:26.401830 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.401834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.402564 (+ 730us) server_negotiation.cc:407] Connection header received
0928 03:01:26.402752 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.402758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.402837 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.402937 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.404421 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.405108 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.406565 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.407126 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.407218 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.411072 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.411077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.411080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.411113 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.413744 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.414491 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.414494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.414497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.414613 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.415976 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.415981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.415984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.416286 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.416424 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.417069 (+ 645us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.417323 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 03:01:26.419261 26674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:01:26 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
Sep 28 03:01:26 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564486, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
Sep 28 03:01:26 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:26.431365 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.416741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.12.182.124:46113)
0928 03:01:26.416887 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.416892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.416905 (+ 13us) server_negotiation.cc:407] Connection header received
0928 03:01:26.416946 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.416950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.417010 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.418066 (+ 1056us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.418195 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.419106 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.420602 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.421145 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.421301 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.424848 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.424852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.424855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.424882 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.427999 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.428622 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.428627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.428631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.428744 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.429073 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.429079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.429083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.430283 (+ 1200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.430430 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.430869 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.431057 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:01:26.431404 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.416116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51172)
0928 03:01:26.416302 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.416320 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:26.416411 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:26.417174 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:26.417179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:26.417674 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:26.417939 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.417953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.419226 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.419231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.420484 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.420496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.421344 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.421351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.421549 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.422160 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:26.422181 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:26.424682 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:26.428156 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.428160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.428164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.428521 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.428847 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.428850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.428853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.428970 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.430430 (+ 1460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:26.430436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:26.430587 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 03:01:26.430754 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 03:01:26.431003 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 03:01:26.434103 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.419933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:43307 (local address 127.0.0.1:43234)
0928 03:01:26.420691 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.420707 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:26.420792 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:26.421136 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:26.421140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:26.421532 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:26.421704 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.421715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.423007 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.423013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.424135 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.424150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.425229 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.425235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.425438 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.425927 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:26.425953 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:26.428205 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:26.431676 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.431680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.431684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.432137 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.432535 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.432540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.432544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.432717 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.433206 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:26.433211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:26.433348 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 03:01:26.433481 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 03:01:26.433753 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661}
I20260928 03:01:26.434158 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.420132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.12.182.125:43307)
0928 03:01:26.420328 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.420334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.420772 (+ 438us) server_negotiation.cc:407] Connection header received
0928 03:01:26.420902 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.420908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.420991 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.421097 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.421887 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.422886 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.424266 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.424924 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.425036 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.428387 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.428394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.428398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.428438 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.431563 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.432257 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.432262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.432265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.432441 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.432846 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.432852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.432855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.433098 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.433230 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.433588 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.433806 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:01:26.437877 26526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51: Bootstrap starting.
I20260928 03:01:26.438295 26674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:01:26.438717 26530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.417671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.12.182.126:35651)
0928 03:01:26.418557 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.418563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.422508 (+ 3945us) server_negotiation.cc:407] Connection header received
0928 03:01:26.422563 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.422568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.422634 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.422722 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.424054 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.425151 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.426820 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.427295 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.427402 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.432523 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.432529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.432534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.432568 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.435986 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.436820 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.436826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.436830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.436966 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.437377 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.437381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.437383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.437712 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.437989 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.438240 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.438437 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":4703,"threads_started":1}
I20260928 03:01:26.442334 26526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:26.443638 26526 log.cc:824] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:01:26 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564486, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
I20260928 03:01:26.446866 26526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51: No bootstrap required, opened a new log
Sep 28 03:01:26 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
I20260928 03:01:26.451779 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.438986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51180)
0928 03:01:26.439126 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.439144 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:26.439230 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:26.439683 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:26.439687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:26.440027 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:26.440184 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.440195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.441269 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.441274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.442152 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.442175 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.442917 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.442923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.443111 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.443690 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:26.443712 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:26.446037 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:26.449510 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.449515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.449519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.449962 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.450271 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.450276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.450279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.450397 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.450895 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:26.450900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:26.451032 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 03:01:26.451143 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 03:01:26.451421 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 03:01:26.451858 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.439160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.12.182.124:46113)
0928 03:01:26.439477 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.439482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.439494 (+ 12us) server_negotiation.cc:407] Connection header received
0928 03:01:26.439525 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.439528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.439574 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.439647 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.440317 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.441118 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.442280 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.442760 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.442883 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.446231 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.446237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.446241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.446273 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.449379 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.450061 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.450066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.450069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.450179 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.450493 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.450498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.450501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.450790 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.450921 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.451267 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.451552 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":225}
I20260928 03:01:26.455744 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.440870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.12.182.125:43307)
0928 03:01:26.441020 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.441025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.441040 (+ 15us) server_negotiation.cc:407] Connection header received
0928 03:01:26.441082 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.441086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.441147 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.441233 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.442760 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.443640 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.445113 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.445544 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.445667 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.449874 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.449880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.449885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.449922 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.453182 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.453943 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.453949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.453952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.454080 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.454435 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.454440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.454444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.454695 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.454948 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.455226 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.455427 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:01:26.457041 26526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } }
I20260928 03:01:26.457422 26526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:26.457535 26526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06afd1cf7ce244459c16e05f0cdd2f51, State: Initialized, Role: FOLLOWER
I20260928 03:01:26.458271 26674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538: Bootstrap starting.
I20260928 03:01:26.458184 26526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } }
I20260928 03:01:26.459039 26681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } } }
I20260928 03:01:26.459626 26681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:01:26.460281 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:01:26.460412 26526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:01:26.462985 26674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:26.464301 26674 log.cc:824] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:26.470701 26674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538: No bootstrap required, opened a new log
I20260928 03:01:26.482892 26674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } }
I20260928 03:01:26.483366 26674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:26.483480 26674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4770476f67dd40c3a585fa01c9de5538, State: Initialized, Role: FOLLOWER
I20260928 03:01:26.484295 26674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } }
I20260928 03:01:26.488505 26691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } } }
I20260928 03:01:26.488912 26691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:01:26.490763 26674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:01:26.499059 26530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.480232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.12.182.126:35651)
0928 03:01:26.480467 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.480473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.481262 (+ 789us) server_negotiation.cc:407] Connection header received
0928 03:01:26.481544 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.481552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.481662 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.481738 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.484118 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.485319 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.487517 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.488275 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.488379 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.492137 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.492146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.492151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.492193 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.495798 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.496675 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.496682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.496686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.496880 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.497326 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.497332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.497335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.497720 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.497938 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.498486 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.498721 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:01:26.511124 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.480951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.12.182.125:43307)
0928 03:01:26.481119 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.481125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.493485 (+ 12360us) server_negotiation.cc:407] Connection header received
0928 03:01:26.494710 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.494716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.494799 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.494879 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.497825 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.498854 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.500987 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.501493 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.501598 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.504032 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.504037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.504040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.504071 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.507101 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.508030 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.508037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.508041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.508231 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.508682 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.508688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.508692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.509067 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.509310 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.510499 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.510745 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:01:26.518002 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.493993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51192 (local address 127.12.182.124:46113)
0928 03:01:26.494157 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.494163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.497985 (+ 3822us) server_negotiation.cc:407] Connection header received
0928 03:01:26.499086 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.499093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.499173 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.499241 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.503702 (+ 4461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.504796 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.506912 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.507634 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.507726 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.511181 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.511188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.511194 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.511235 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.515011 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.515996 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.516002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.516005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.516159 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.516528 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.516533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.516536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.516894 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.517156 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.517412 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.517657 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 03:01:26.528573 26710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:01:26.528714 26710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:01:26.523825 26459 master_runner.cc:186] Error getting master registration for 127.12.182.125:43307: OK, instance_id { permanent_uuid: "1e87781cb5424fad937006a16697cc27" instance_seqno: 1790564485906914 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:01:26.533931 26459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 03:01:26.537413 26713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:01:26.537539 26713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:01:26.536123 26607 master_runner.cc:186] Error getting master registration for 127.12.182.125:43307: OK, instance_id { permanent_uuid: "1e87781cb5424fad937006a16697cc27" instance_seqno: 1790564485906914 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:01:26.547032 26607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 03:01:26.621033 26628 tablet.cc:2369] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:01:26.626094 26600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } attempt: 5
Sep 28 03:01:26 dist-test-slave-k5rg krb5kdc[26446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
I20260928 03:01:26.642481 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.626624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:46113 (local address 127.0.0.1:51202)
0928 03:01:26.626868 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:26.626889 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:26.627000 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:26.627338 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:26.627343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:26.627844 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:26.628043 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.628056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.629460 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.629466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.630902 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:26.630919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.631881 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.631889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:26.632123 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.632782 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:26.632804 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:26.635558 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:26.639589 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.639595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.639599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.640099 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.640499 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:26.640505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:26.640509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:26.640712 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:26.641467 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:26.641472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:26.641646 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 03:01:26.641787 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 03:01:26.642096 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 03:01:26.643477 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:26.626697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.12.182.124:46113)
0928 03:01:26.626884 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:26.626890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:26.626939 (+ 49us) server_negotiation.cc:407] Connection header received
0928 03:01:26.627109 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:26.627115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:26.627196 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:26.627294 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:26.628192 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.629309 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.631057 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:26.631732 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:26.631866 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:26.635740 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:26.635747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:26.635753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:26.635797 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:26.639465 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.640217 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.640223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.640227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.640386 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:26.640863 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:26.640868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:26.640872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:26.641241 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:26.641401 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:26.642824 (+ 1423us) server_negotiation.cc:299] Negotiation successful
0928 03:01:26.643089 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 03:01:26.649377 26600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27: Bootstrap starting.
I20260928 03:01:26.653213 26600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:26.654474 26600 log.cc:824] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:26.657491 26600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27: No bootstrap required, opened a new log
I20260928 03:01:26.668145 26600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } }
I20260928 03:01:26.668514 26600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:26.668625 26600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87781cb5424fad937006a16697cc27, State: Initialized, Role: FOLLOWER
I20260928 03:01:26.669255 26600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } }
I20260928 03:01:26.670238 26716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } } }
I20260928 03:01:26.670704 26716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:01:26.671424 26600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:01:26.684433 26533 master_runner.cc:263] No leader master found from master 1e87781cb5424fad937006a16697cc27
I20260928 03:01:26.684566 26533 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 03:01:26.687176 26727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:01:26.687393 26727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 03:01:26.697714 26480 tablet.cc:2369] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:01:26.741176 26716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:01:26.741503 26716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } }
I20260928 03:01:26.743182 26716 leader_election.cc:288] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06afd1cf7ce244459c16e05f0cdd2f51 (127.12.182.126:35651), 4770476f67dd40c3a585fa01c9de5538 (127.12.182.124:46113)
I20260928 03:01:26.744314 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e87781cb5424fad937006a16697cc27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4770476f67dd40c3a585fa01c9de5538" is_pre_election: true
I20260928 03:01:26.744943 26649 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87781cb5424fad937006a16697cc27 in term 0.
I20260928 03:01:26.746153 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e87781cb5424fad937006a16697cc27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" is_pre_election: true
I20260928 03:01:26.746810 26501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87781cb5424fad937006a16697cc27 in term 0.
I20260928 03:01:26.747666 26551 leader_election.cc:302] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87781cb5424fad937006a16697cc27, 4770476f67dd40c3a585fa01c9de5538; no voters:
I20260928 03:01:26.752419 26716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:01:26.752576 26716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:01:26.752674 26716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:26.756644 26716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } }
I20260928 03:01:26.757874 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e87781cb5424fad937006a16697cc27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06afd1cf7ce244459c16e05f0cdd2f51"
I20260928 03:01:26.761512 26501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:26.765024 26501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87781cb5424fad937006a16697cc27 in term 1.
I20260928 03:01:26.765769 26551 leader_election.cc:302] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06afd1cf7ce244459c16e05f0cdd2f51, 1e87781cb5424fad937006a16697cc27; no voters:
I20260928 03:01:26.768232 26716 leader_election.cc:288] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [CANDIDATE]: Term 1 election: Requested vote from peers 06afd1cf7ce244459c16e05f0cdd2f51 (127.12.182.126:35651), 4770476f67dd40c3a585fa01c9de5538 (127.12.182.124:46113)
I20260928 03:01:26.768818 26716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:01:26.769237 26716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [term 1 LEADER]: Becoming Leader. State: Replica: 1e87781cb5424fad937006a16697cc27, State: Running, Role: LEADER
I20260928 03:01:26.770123 26716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } }
I20260928 03:01:26.784024 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e87781cb5424fad937006a16697cc27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4770476f67dd40c3a585fa01c9de5538"
I20260928 03:01:26.784276 26649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:26.795303 26649 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87781cb5424fad937006a16697cc27 in term 1.
I20260928 03:01:26.801196 26729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e87781cb5424fad937006a16697cc27. Latest consensus state: current_term: 1 leader_uuid: "1e87781cb5424fad937006a16697cc27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } } }
I20260928 03:01:26.801751 26729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:26.814173 26731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:01:26.816474 26731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:01:26.831118 26649 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [term 1 FOLLOWER]: Refusing update from remote peer 1e87781cb5424fad937006a16697cc27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:01:26.832077 26729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:01:26.839259 26501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [term 1 FOLLOWER]: Refusing update from remote peer 1e87781cb5424fad937006a16697cc27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:01:26.840045 26729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:01:26.874780 26681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e87781cb5424fad937006a16697cc27. Latest consensus state: current_term: 1 leader_uuid: "1e87781cb5424fad937006a16697cc27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } } }
I20260928 03:01:26.875093 26681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:01:26.881659 26691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e87781cb5424fad937006a16697cc27. Latest consensus state: current_term: 1 leader_uuid: "1e87781cb5424fad937006a16697cc27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } } }
I20260928 03:01:26.882006 26691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:01:26.883703 26734 mvcc.cc:205] Tried to move back new op lower bound from 7334152138049683456 to 7334152137824993280. Current Snapshot: MvccSnapshot[applied={T|T < 7334152138049683456}]
I20260928 03:01:26.885514 26691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e87781cb5424fad937006a16697cc27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } } }
I20260928 03:01:26.885768 26691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:01:26.890396 26681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e87781cb5424fad937006a16697cc27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } } }
I20260928 03:01:26.890645 26681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:01:26.893070 26729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e87781cb5424fad937006a16697cc27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } } }
I20260928 03:01:26.893378 26729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:26.893784 26729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e87781cb5424fad937006a16697cc27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06afd1cf7ce244459c16e05f0cdd2f51" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35651 } } peers { permanent_uuid: "1e87781cb5424fad937006a16697cc27" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 43307 } } peers { permanent_uuid: "4770476f67dd40c3a585fa01c9de5538" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 46113 } } }
I20260928 03:01:26.894013 26729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:26.894657 26731 catalog_manager.cc:1347] Generated new cluster ID: 3b1e265d48eb44aca96cb6c50d89bb44
I20260928 03:01:26.894753 26731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:01:26.943547 26731 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:01:26.950306 26731 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:01:26.969225 26731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e87781cb5424fad937006a16697cc27: Generated new TSK 0
I20260928 03:01:26.970264 26731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:01:27.530714 26710 catalog_manager.cc:1259] Loaded cluster ID: 3b1e265d48eb44aca96cb6c50d89bb44
I20260928 03:01:27.530814 26710 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51: loading cluster ID for follower catalog manager: success
I20260928 03:01:27.533979 26710 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51: acquiring CA information for follower catalog manager: success
I20260928 03:01:27.535730 26710 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06afd1cf7ce244459c16e05f0cdd2f51: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:01:27.539166 26713 catalog_manager.cc:1259] Loaded cluster ID: 3b1e265d48eb44aca96cb6c50d89bb44
I20260928 03:01:27.539242 26713 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538: loading cluster ID for follower catalog manager: success
I20260928 03:01:27.541388 26713 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538: acquiring CA information for follower catalog manager: success
I20260928 03:01:27.542970 26713 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4770476f67dd40c3a585fa01c9de5538: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:01:28.049811 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 26459
I20260928 03:01:28.051079 26521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:28.159970 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 26459
I20260928 03:01:28.176821 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 26533
I20260928 03:01:28.177910 26595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:28.282320 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 26533
I20260928 03:01:28.298362 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 26607
I20260928 03:01:28.299386 26669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:28.403681 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 26607
2026-09-28T03:01:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4597 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:01:28 dist-test-slave-k5rg krb5kdc[26750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:01:28 dist-test-slave-k5rg krb5kdc[26750](info): set up 2 sockets
Sep 28 03:01:28 dist-test-slave-k5rg krb5kdc[26750](info): commencing operation
krb5kdc: starting...
W20260928 03:01:29.409965 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:01:29 dist-test-slave-k5rg krb5kdc[26750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:01:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:01:29.618718 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:38349
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:38349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:29.929734 26763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:29.930059 26763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:29.930193 26763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:29.937561 26763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:01:29.937685 26763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:29.937739 26763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:29.937819 26763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:01:29.937875 26763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:01:29.948184 26763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:38349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:38349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:29.949510 26763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:29.950817 26763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:29.962006 26768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:29.961982 26769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:29.961915 26771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:29.962877 26763 server_base.cc:1034] running on GCE node
I20260928 03:01:29.963589 26763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:29.964926 26763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:29.966114 26763 hybrid_clock.cc:648] HybridClock initialized: now 1790564489966059 us; error 48 us; skew 500 ppm
I20260928 03:01:29.966430 26763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:29 dist-test-slave-k5rg krb5kdc[26750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564489, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:29.969331 26763 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:01:29.970723 26763 webserver.cc:460] Webserver started at http://127.12.182.126:46799/ using document root <none> and password file <none>
I20260928 03:01:29.971380 26763 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:29.971480 26763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:29.971746 26763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:29.974654 26763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "6e634048c13c411d95a4d3a832ebea17"
format_stamp: "Formatted at 2026-09-28 03:01:29 on dist-test-slave-k5rg"
server_key: "002857706b2f23c3f7dbaed7c619dc24"
server_key_iv: "6501af92081bbb65bcad5a508e62408b"
server_key_version: "encryptionkey@0"
I20260928 03:01:29.975338 26763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "6e634048c13c411d95a4d3a832ebea17"
format_stamp: "Formatted at 2026-09-28 03:01:29 on dist-test-slave-k5rg"
server_key: "002857706b2f23c3f7dbaed7c619dc24"
server_key_iv: "6501af92081bbb65bcad5a508e62408b"
server_key_version: "encryptionkey@0"
I20260928 03:01:29.980216 26763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:01:29.983618 26778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:29.984972 26763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:01:29.985113 26763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "6e634048c13c411d95a4d3a832ebea17"
format_stamp: "Formatted at 2026-09-28 03:01:29 on dist-test-slave-k5rg"
server_key: "002857706b2f23c3f7dbaed7c619dc24"
server_key_iv: "6501af92081bbb65bcad5a508e62408b"
server_key_version: "encryptionkey@0"
I20260928 03:01:29.985277 26763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:30.003412 26763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:30.007642 26763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:30.007921 26763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:30.025187 26763 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:38349
I20260928 03:01:30.025195 26829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:38349 every 8 connection(s)
I20260928 03:01:30.026535 26763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:01:30.028450 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 26763
I20260928 03:01:30.028638 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:01:30.029237 13017 external_mini_cluster.cc:1428] Setting key 2a027d5a410509e9ddf184fdec33f60e
I20260928 03:01:30.030284 26830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 03:01:30 dist-test-slave-k5rg krb5kdc[26750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:30.043828 26830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17: Bootstrap starting.
I20260928 03:01:30.047043 26830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:30.048218 26830 log.cc:824] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:30.051424 26830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17: No bootstrap required, opened a new log
I20260928 03:01:30.051575 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:30.030968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.12.182.126:38349)
0928 03:01:30.031682 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:30.031693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:30.031732 (+ 39us) server_negotiation.cc:407] Connection header received
0928 03:01:30.033302 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:30.033322 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:30.033719 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:30.034046 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:30.035789 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:30.037116 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:30.038926 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:30.039567 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:30.039702 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:30.044382 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:30.044390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:30.044403 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:30.044438 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:30.047307 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:30.048059 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:30.048065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:30.048070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:30.048226 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:30.048603 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:30.048606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:30.048608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:30.049333 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:30.049629 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:30.050020 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 03:01:30.050448 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":330,"threads_started":1}
I20260928 03:01:30.055286 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:01:30.059204 26830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e634048c13c411d95a4d3a832ebea17" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38349 } }
I20260928 03:01:30.059527 26830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:30.059617 26830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e634048c13c411d95a4d3a832ebea17, State: Initialized, Role: FOLLOWER
I20260928 03:01:30.060134 26830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e634048c13c411d95a4d3a832ebea17" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38349 } }
I20260928 03:01:30.060338 26830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:01:30.060462 26830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:01:30.060600 26830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:30.062673 26830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e634048c13c411d95a4d3a832ebea17" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38349 } }
I20260928 03:01:30.063128 26830 leader_election.cc:302] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e634048c13c411d95a4d3a832ebea17; no voters:
I20260928 03:01:30.063647 26830 leader_election.cc:288] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:01:30.063783 26835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:01:30.064414 26835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [term 1 LEADER]: Becoming Leader. State: Replica: 6e634048c13c411d95a4d3a832ebea17, State: Running, Role: LEADER
I20260928 03:01:30.064895 26835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e634048c13c411d95a4d3a832ebea17" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38349 } }
I20260928 03:01:30.065541 26830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:01:30.067492 26836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e634048c13c411d95a4d3a832ebea17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e634048c13c411d95a4d3a832ebea17" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38349 } } }
I20260928 03:01:30.067466 26837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e634048c13c411d95a4d3a832ebea17. Latest consensus state: current_term: 1 leader_uuid: "6e634048c13c411d95a4d3a832ebea17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e634048c13c411d95a4d3a832ebea17" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38349 } } }
I20260928 03:01:30.067929 26836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:30.068045 26837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:30.069576 26841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:01:30.074354 26841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:01:30.082973 26841 catalog_manager.cc:1347] Generated new cluster ID: 6912a84913c64c29954ae362e9b6ac8e
I20260928 03:01:30.083081 26841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:01:30.090252 26841 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:01:30.092020 26841 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:01:30.105402 26841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e634048c13c411d95a4d3a832ebea17: Generated new TSK 0
I20260928 03:01:30.106531 26841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:01:30.115635 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 26763
I20260928 03:01:30.116773 26825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:30.204574 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 26763
2026-09-28T03:01:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1796 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:01:30 dist-test-slave-k5rg krb5kdc[26858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:01:30 dist-test-slave-k5rg krb5kdc[26858](info): set up 2 sockets
Sep 28 03:01:30 dist-test-slave-k5rg krb5kdc[26858](info): commencing operation
krb5kdc: starting...
W20260928 03:01:31.205948 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:01:31 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:01:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:01:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:01:31.422065 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:37675
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:37675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:31.723199 26871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:31.723474 26871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:31.723608 26871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:31.730520 26871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:01:31.730620 26871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:31.730698 26871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:31.730777 26871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:01:31.730839 26871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:01:31.740042 26871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:37675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:31.741228 26871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:31.742448 26871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:31.752988 26876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:31.753010 26879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:31.753050 26877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:31.754314 26871 server_base.cc:1034] running on GCE node
I20260928 03:01:31.755004 26871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:31.756291 26871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:31.757508 26871 hybrid_clock.cc:648] HybridClock initialized: now 1790564491757468 us; error 30 us; skew 500 ppm
I20260928 03:01:31.757831 26871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:31 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564491, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:31.764926 26871 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:01:31.766374 26871 webserver.cc:460] Webserver started at http://127.12.182.126:32827/ using document root <none> and password file <none>
I20260928 03:01:31.766983 26871 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:31.767093 26871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:31.767369 26871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:31.770118 26871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "c5993e2bd9b24067a036df645c9b3c55"
format_stamp: "Formatted at 2026-09-28 03:01:31 on dist-test-slave-k5rg"
server_key: "964eaaac5861ff4166696ffe8647e1ea"
server_key_iv: "05d8d3a8b57581a34f3126779ff23727"
server_key_version: "encryptionkey@0"
I20260928 03:01:31.770821 26871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "c5993e2bd9b24067a036df645c9b3c55"
format_stamp: "Formatted at 2026-09-28 03:01:31 on dist-test-slave-k5rg"
server_key: "964eaaac5861ff4166696ffe8647e1ea"
server_key_iv: "05d8d3a8b57581a34f3126779ff23727"
server_key_version: "encryptionkey@0"
I20260928 03:01:31.775672 26871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 03:01:31.779279 26886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:31.780503 26871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 03:01:31.780646 26871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "c5993e2bd9b24067a036df645c9b3c55"
format_stamp: "Formatted at 2026-09-28 03:01:31 on dist-test-slave-k5rg"
server_key: "964eaaac5861ff4166696ffe8647e1ea"
server_key_iv: "05d8d3a8b57581a34f3126779ff23727"
server_key_version: "encryptionkey@0"
I20260928 03:01:31.780844 26871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:31.797858 26871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:31.801487 26871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:31.801748 26871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:31.817588 26871 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:37675
I20260928 03:01:31.817642 26937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:37675 every 8 connection(s)
I20260928 03:01:31.818851 26871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:01:31.822222 26938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:31.829180 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 26871
I20260928 03:01:31.829397 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:01:31.830195 13017 external_mini_cluster.cc:1428] Setting key bc648086724bd56b4c4345d4ac6dcbc0
I20260928 03:01:31.832686 26938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Bootstrap starting.
I20260928 03:01:31.835599 26938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:31.836493 26938 log.cc:824] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:31.839037 26938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: No bootstrap required, opened a new log
Sep 28 03:01:31 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:31.846524 26938 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:31.846865 26938 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:31.846975 26938 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5993e2bd9b24067a036df645c9b3c55, State: Initialized, Role: FOLLOWER
I20260928 03:01:31.847506 26938 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:31.847721 26938 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:01:31.847833 26938 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:01:31.847977 26938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:31.850277 26938 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:31.850713 26938 leader_election.cc:302] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5993e2bd9b24067a036df645c9b3c55; no voters:
I20260928 03:01:31.850878 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:31.832121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.12.182.126:37675)
0928 03:01:31.832851 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:31.832865 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:31.832905 (+ 40us) server_negotiation.cc:407] Connection header received
0928 03:01:31.834558 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:31.834579 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:31.834952 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:31.835275 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:31.836615 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:31.837939 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:31.839175 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:31.839823 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:31.839993 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:31.843978 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:31.843988 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:31.844004 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:31.844042 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:31.846840 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:31.847499 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:31.847505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:31.847510 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:31.847661 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:31.848085 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:31.848090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:31.848092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:31.848607 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:31.849001 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:31.849426 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 03:01:31.849880 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":316,"threads_started":1}
I20260928 03:01:31.852487 26938 leader_election.cc:288] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:01:31.852594 26943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:01:31.853948 26943 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 1 LEADER]: Becoming Leader. State: Replica: c5993e2bd9b24067a036df645c9b3c55, State: Running, Role: LEADER
I20260928 03:01:31.854519 26943 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:31.854734 26938 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:01:31.857769 26945 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5993e2bd9b24067a036df645c9b3c55. Latest consensus state: current_term: 1 leader_uuid: "c5993e2bd9b24067a036df645c9b3c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } } }
I20260928 03:01:31.857829 26944 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5993e2bd9b24067a036df645c9b3c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } } }
I20260928 03:01:31.858179 26945 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:31.858186 26944 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:31.861020 26949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:01:31.865319 26949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:01:31.872603 26949 catalog_manager.cc:1347] Generated new cluster ID: 837f37c7c79b4744bea5d96e284d23d9
I20260928 03:01:31.872699 26949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:01:31.879781 26949 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:01:31.881534 26949 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:01:31.888814 26949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Generated new TSK 0
I20260928 03:01:31.889719 26949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 03:01:31.932636 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:37675
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:32.239800 26964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:32.240057 26964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:32.240185 26964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:32.247109 26964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:32.247200 26964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:32.247342 26964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:01:32.256780 26964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:32.257992 26964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:32.259181 26964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:32.270919 26969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:32.270983 26970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:32.270946 26972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:32.272058 26964 server_base.cc:1034] running on GCE node
I20260928 03:01:32.272584 26964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:32.273391 26964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:32.274576 26964 hybrid_clock.cc:648] HybridClock initialized: now 1790564492274533 us; error 30 us; skew 500 ppm
I20260928 03:01:32.274866 26964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:32 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564492, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:32.277684 26964 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:01:32.279042 26964 webserver.cc:460] Webserver started at http://127.12.182.65:40563/ using document root <none> and password file <none>
I20260928 03:01:32.279657 26964 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:32.279767 26964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:32.280037 26964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:32.282707 26964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "1db6101237d2491bb46fb86536f634d3"
format_stamp: "Formatted at 2026-09-28 03:01:32 on dist-test-slave-k5rg"
server_key: "e09f2ee71aed04288040663e956489fc"
server_key_iv: "a83c60281eee02ea1f73c5b6475cc138"
server_key_version: "encryptionkey@0"
I20260928 03:01:32.283417 26964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "1db6101237d2491bb46fb86536f634d3"
format_stamp: "Formatted at 2026-09-28 03:01:32 on dist-test-slave-k5rg"
server_key: "e09f2ee71aed04288040663e956489fc"
server_key_iv: "a83c60281eee02ea1f73c5b6475cc138"
server_key_version: "encryptionkey@0"
I20260928 03:01:32.288195 26964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 03:01:32.291370 26979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:32.292543 26964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 03:01:32.292685 26964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "1db6101237d2491bb46fb86536f634d3"
format_stamp: "Formatted at 2026-09-28 03:01:32 on dist-test-slave-k5rg"
server_key: "e09f2ee71aed04288040663e956489fc"
server_key_iv: "a83c60281eee02ea1f73c5b6475cc138"
server_key_version: "encryptionkey@0"
I20260928 03:01:32.292857 26964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:32.305162 26964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:32.308522 26964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:32.308760 26964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:32.309576 26964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:32.310828 26964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:32.310909 26964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:32.311019 26964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:32.311069 26964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:32.339183 26964 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:44907
I20260928 03:01:32.339221 27091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:44907 every 8 connection(s)
I20260928 03:01:32.340361 26964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:01:32.341552 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 26964
I20260928 03:01:32.341789 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 03:01:32.342550 13017 external_mini_cluster.cc:1428] Setting key cab504cd30c72e02aa6a4c14bf4ea3d6
Sep 28 03:01:32 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564492, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:32.360646 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:32.342990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:39367 (local address 127.12.182.126:37675)
0928 03:01:32.343195 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:32.343202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:32.344413 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 03:01:32.346294 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:32.346300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:32.346376 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:32.346472 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:32.347856 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:32.348854 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:32.350443 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:32.350836 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:32.350977 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:32.354996 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:32.355021 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:32.355025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:32.355058 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:32.357625 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:32.358389 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:32.358392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:32.358394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:32.358509 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:32.358923 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:32.358928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:32.358930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:32.359236 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:32.359372 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:32.360006 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 03:01:32.360255 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:01:32.361500 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:32.343486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:39367)
0928 03:01:32.344253 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:32.344302 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:32.346043 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:32.346649 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:32.346656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:32.347132 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:32.347695 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:32.347712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:32.349020 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:32.349026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:32.350307 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:32.350319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:32.351112 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:32.351120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:32.351433 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:32.352162 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:32.352189 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:32.354764 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:32.357765 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:32.357773 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:32.357787 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:32.358287 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:32.358614 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:32.358619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:32.358622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:32.358803 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:32.359362 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:32.359369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:32.359665 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 03:01:32.359890 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 03:01:32.360374 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":300,"threads_started":1}
I20260928 03:01:32.362996 27092 heartbeater.cc:347] Connected to a master server at 127.12.182.126:37675
I20260928 03:01:32.363332 27092 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
I20260928 03:01:32.364230 27092 heartbeater.cc:511] Master 127.12.182.126:37675 requested a full tablet report, sending...
I20260928 03:01:32.366196 26903 ts_manager.cc:194] Registered new tserver with Master: 1db6101237d2491bb46fb86536f634d3 (127.12.182.65:44907)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
I20260928 03:01:32.367635 26903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:39367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 03:01:32.390128 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:37675
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:32.692710 27097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:32.692972 27097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:32.693065 27097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:32.700152 27097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:32.700253 27097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:32.700371 27097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:01:32.709745 27097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:32.710891 27097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:32.712092 27097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:32.723892 27102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:32.723933 27105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:32.723977 27103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:32.724926 27097 server_base.cc:1034] running on GCE node
I20260928 03:01:32.725486 27097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:32.726351 27097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:32.727545 27097 hybrid_clock.cc:648] HybridClock initialized: now 1790564492727494 us; error 45 us; skew 500 ppm
I20260928 03:01:32.727828 27097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:32 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564492, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:32.730762 27097 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:01:32.732071 27097 webserver.cc:460] Webserver started at http://127.12.182.66:44137/ using document root <none> and password file <none>
I20260928 03:01:32.732687 27097 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:32.732781 27097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:32.733011 27097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:32.735749 27097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0"
format_stamp: "Formatted at 2026-09-28 03:01:32 on dist-test-slave-k5rg"
server_key: "1e02d12110499b268f32676e45693079"
server_key_iv: "a102abdd031747b469e2ff7c89ab46e6"
server_key_version: "encryptionkey@0"
I20260928 03:01:32.736452 27097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0"
format_stamp: "Formatted at 2026-09-28 03:01:32 on dist-test-slave-k5rg"
server_key: "1e02d12110499b268f32676e45693079"
server_key_iv: "a102abdd031747b469e2ff7c89ab46e6"
server_key_version: "encryptionkey@0"
I20260928 03:01:32.741186 27097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 03:01:32.744357 27112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:32.745568 27097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 03:01:32.745757 27097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0"
format_stamp: "Formatted at 2026-09-28 03:01:32 on dist-test-slave-k5rg"
server_key: "1e02d12110499b268f32676e45693079"
server_key_iv: "a102abdd031747b469e2ff7c89ab46e6"
server_key_version: "encryptionkey@0"
I20260928 03:01:32.745930 27097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:32.781016 27097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:32.785151 27097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:32.785414 27097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:32.786434 27097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:32.787988 27097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:32.788098 27097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:32.788233 27097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:32.788291 27097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:32.819787 27097 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:37375
I20260928 03:01:32.819826 27224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:37375 every 8 connection(s)
I20260928 03:01:32.820938 27097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:01:32.829113 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 27097
I20260928 03:01:32.829325 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 03:01:32.829891 13017 external_mini_cluster.cc:1428] Setting key 3428fb0b3a63b10ca5184d446f431a53
Sep 28 03:01:32 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564492, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:32.842010 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:32.823351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:57033 (local address 127.12.182.126:37675)
0928 03:01:32.823544 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:32.823550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:32.824719 (+ 1169us) server_negotiation.cc:407] Connection header received
0928 03:01:32.827152 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:32.827156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:32.827240 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:32.827320 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:32.829140 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:32.830209 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:32.832421 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:32.833064 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:32.833182 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:32.836594 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:32.836599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:32.836602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:32.836634 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:32.839196 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:32.839871 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:32.839876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:32.839879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:32.840023 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:32.840364 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:32.840368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:32.840372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:32.840681 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:32.840833 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:32.841471 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 03:01:32.841705 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 03:01:32.842832 27227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:32.823820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:57033)
0928 03:01:32.824553 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:32.824593 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:32.826852 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:32.827576 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:32.827587 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:32.828241 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:32.828954 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:32.828976 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:32.830403 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:32.830409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:32.832267 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:32.832281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:32.833281 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:32.833288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:32.833560 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:32.834274 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:32.834307 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:32.836391 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:32.839339 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:32.839345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:32.839357 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:32.839747 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:32.840111 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:32.840114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:32.840116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:32.840277 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:32.840838 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:32.840845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:32.841138 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 03:01:32.841365 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 03:01:32.841778 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":266,"threads_started":1}
I20260928 03:01:32.844245 27225 heartbeater.cc:347] Connected to a master server at 127.12.182.126:37675
I20260928 03:01:32.844511 27225 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:32.845183 27225 heartbeater.cc:511] Master 127.12.182.126:37675 requested a full tablet report, sending...
I20260928 03:01:32.846527 26903 ts_manager.cc:194] Registered new tserver with Master: f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375)
I20260928 03:01:32.847338 26903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:57033
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 03:01:32.876988 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:37675
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:33.176872 27230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:33.177145 27230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:33.177268 27230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:33.184444 27230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:33.184552 27230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:33.184698 27230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:01:33.194347 27230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:33.195616 27230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:33.196879 27230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:33.208986 27235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:33.208947 27238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:33.209019 27236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:33.209961 27230 server_base.cc:1034] running on GCE node
I20260928 03:01:33.210480 27230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:33.211297 27230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:33.212486 27230 hybrid_clock.cc:648] HybridClock initialized: now 1790564493212436 us; error 44 us; skew 500 ppm
I20260928 03:01:33.212759 27230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:33 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564493, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:33.215626 27230 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:01:33.216862 27230 webserver.cc:460] Webserver started at http://127.12.182.67:32865/ using document root <none> and password file <none>
I20260928 03:01:33.217470 27230 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:33.217578 27230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:33.217869 27230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:33.220536 27230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "7a7148b3104c42138425b9b052484de5"
format_stamp: "Formatted at 2026-09-28 03:01:33 on dist-test-slave-k5rg"
server_key: "5709d05926ca05aee62759d615355b78"
server_key_iv: "7217f16bef8198b9a2e0a3d230826a27"
server_key_version: "encryptionkey@0"
I20260928 03:01:33.221220 27230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "7a7148b3104c42138425b9b052484de5"
format_stamp: "Formatted at 2026-09-28 03:01:33 on dist-test-slave-k5rg"
server_key: "5709d05926ca05aee62759d615355b78"
server_key_iv: "7217f16bef8198b9a2e0a3d230826a27"
server_key_version: "encryptionkey@0"
I20260928 03:01:33.225878 27230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260928 03:01:33.229060 27245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:33.230271 27230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 03:01:33.230414 27230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "7a7148b3104c42138425b9b052484de5"
format_stamp: "Formatted at 2026-09-28 03:01:33 on dist-test-slave-k5rg"
server_key: "5709d05926ca05aee62759d615355b78"
server_key_iv: "7217f16bef8198b9a2e0a3d230826a27"
server_key_version: "encryptionkey@0"
I20260928 03:01:33.230581 27230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:33.250138 27230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:33.254070 27230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:33.254323 27230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:33.255241 27230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:33.256588 27230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:33.256668 27230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:33.256786 27230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:33.256837 27230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:33.286136 27230 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:42513
I20260928 03:01:33.286172 27357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:42513 every 8 connection(s)
I20260928 03:01:33.287325 27230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:01:33.294989 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 27230
I20260928 03:01:33.295173 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 03:01:33.295611 13017 external_mini_cluster.cc:1428] Setting key 7d23fa730ce02f84cc0d73fc3f1f7152
Sep 28 03:01:33 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564493, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:33.305332 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.289572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:54675 (local address 127.12.182.126:37675)
0928 03:01:33.289745 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:33.289749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:33.290846 (+ 1097us) server_negotiation.cc:407] Connection header received
0928 03:01:33.292560 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:33.292566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:33.292642 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:33.292727 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:33.294187 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.295072 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.296664 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.297111 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.297225 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.300518 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:33.300523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:33.300526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:33.300558 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:33.302856 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:33.303473 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:33.303477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:33.303479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:33.303588 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:33.303927 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:33.303929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:33.303931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:33.304188 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:33.304296 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:33.304936 (+ 640us) server_negotiation.cc:299] Negotiation successful
0928 03:01:33.305112 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:01:33.306375 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.290049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:54675)
0928 03:01:33.290703 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:33.290738 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:33.292306 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:33.292963 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:33.292971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:33.293447 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:33.294019 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.294037 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.295229 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.295233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.296521 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.296534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.297322 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.297329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.297548 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.298270 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:33.298297 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:33.300323 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:33.302977 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:33.302983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:33.302994 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:33.303377 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:33.303659 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:33.303661 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:33.303664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:33.303845 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:33.304323 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:33.304330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:33.304632 (+ 302us) client_negotiation.cc:769] Sending connection context
0928 03:01:33.304868 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 03:01:33.305231 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":247,"threads_started":1}
I20260928 03:01:33.307577 27358 heartbeater.cc:347] Connected to a master server at 127.12.182.126:37675
I20260928 03:01:33.307858 27358 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:33.308558 27358 heartbeater.cc:511] Master 127.12.182.126:37675 requested a full tablet report, sending...
I20260928 03:01:33.310133 26903 ts_manager.cc:194] Registered new tserver with Master: 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513)
I20260928 03:01:33.310905 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:01:33.311189 26903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:54675
I20260928 03:01:33.327231 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.315760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.12.182.126:37675)
0928 03:01:33.315944 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:33.315950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:33.316345 (+ 395us) server_negotiation.cc:407] Connection header received
0928 03:01:33.316473 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:33.316476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:33.316533 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:33.316603 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:33.317898 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.318585 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.319908 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.320280 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.320370 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.322634 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:33.322639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:33.322642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:33.322671 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:33.324968 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:33.325550 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:33.325553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:33.325555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:33.325699 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:33.325996 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:33.325999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:33.326001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:33.326294 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:33.326477 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:33.326768 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 03:01:33.326949 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 03:01:33.330771 26903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:01:33.337394 26903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:01:33.364807 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.349023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:42513 (local address 127.0.0.1:32986)
0928 03:01:33.356996 (+ 7973us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:33.357014 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:33.357135 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:33.358135 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:33.358142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:33.358184 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:33.358641 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.358653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.360375 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.360380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.362565 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.362582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.363820 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.363830 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.364006 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.364057 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 03:01:33.364163 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 03:01:33.364364 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7835,"mutex_wait_us":1,"thread_start_us":6481,"threads_started":1}
I20260928 03:01:33.365443 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.349738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:37375 (local address 127.0.0.1:32824)
0928 03:01:33.357131 (+ 7393us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:33.357144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:33.357226 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:33.358429 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:33.358434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:33.358456 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:33.358851 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.358863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.360592 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.360597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.362667 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.362682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.364675 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.364681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.364866 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.364898 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 03:01:33.364984 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 03:01:33.365108 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7292,"thread_start_us":8361,"threads_started":1}
I20260928 03:01:33.365562 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.349595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.12.182.67:42513)
0928 03:01:33.356725 (+ 7130us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:33.356734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:33.357263 (+ 529us) server_negotiation.cc:407] Connection header received
0928 03:01:33.357384 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:33.357391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:33.357660 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:33.357839 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:33.358848 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.360241 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.362721 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.363615 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.363728 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.365033 (+ 1305us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:33.365095 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 03:01:33.365244 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6951,"thread_start_us":6789,"threads_started":1}
I20260928 03:01:33.366029 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.350310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.12.182.66:37375)
0928 03:01:33.357648 (+ 7338us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:33.357655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:33.357681 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:01:33.357789 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:33.357797 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:33.358000 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:33.358169 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:33.359034 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.360437 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.362818 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.364208 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.364340 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.364527 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:33.365547 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0928 03:01:33.365742 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7172,"thread_start_us":7102,"threads_started":1}
I20260928 03:01:33.366852 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.348869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.0.0.1:54348)
0928 03:01:33.355883 (+ 7014us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:33.355920 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:33.356095 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:33.359772 (+ 3677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:33.359777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:33.359803 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:33.360169 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.360181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.362302 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.362307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.364357 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.364374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.366159 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.366165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.366323 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.366354 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 03:01:33.366432 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 03:01:33.366545 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6769,"thread_start_us":7841,"threads_started":1}
I20260928 03:01:33.366895 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.349087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.12.182.65:44907)
0928 03:01:33.359251 (+ 10164us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:33.359258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:33.359293 (+ 35us) server_negotiation.cc:407] Connection header received
0928 03:01:33.359377 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:33.359384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:33.359584 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:33.359753 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:33.360530 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.362173 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.364542 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.366002 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.366136 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.366330 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:33.366467 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 03:01:33.366613 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9992,"thread_start_us":9846,"threads_started":1}
I20260928 03:01:33.369457 27160 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf97262abde420781031b4d95eb57b9 (DEFAULT_TABLE table=test-table [id=3481e224f0e44581b3d70a5b61be9ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:33.369455 27293 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf97262abde420781031b4d95eb57b9 (DEFAULT_TABLE table=test-table [id=3481e224f0e44581b3d70a5b61be9ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:33.370325 27092 heartbeater.cc:503] Master 127.12.182.126:37675 was elected leader, sending a full tablet report...
I20260928 03:01:33.370031 27027 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf97262abde420781031b4d95eb57b9 (DEFAULT_TABLE table=test-table [id=3481e224f0e44581b3d70a5b61be9ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:33.370900 27293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf97262abde420781031b4d95eb57b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:33.371300 27027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf97262abde420781031b4d95eb57b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:33.371400 27160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf97262abde420781031b4d95eb57b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:33.381814 27376 tablet_bootstrap.cc:492] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: Bootstrap starting.
I20260928 03:01:33.384750 27377 tablet_bootstrap.cc:492] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: Bootstrap starting.
I20260928 03:01:33.385375 27376 tablet_bootstrap.cc:654] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:33.386190 27378 tablet_bootstrap.cc:492] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: Bootstrap starting.
I20260928 03:01:33.387178 27376 log.cc:824] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:33.388708 27377 tablet_bootstrap.cc:654] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:33.389061 27378 tablet_bootstrap.cc:654] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:33.389891 27377 log.cc:824] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:33.389966 27378 log.cc:824] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:33.390280 27376 tablet_bootstrap.cc:492] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: No bootstrap required, opened a new log
I20260928 03:01:33.390588 27376 ts_tablet_manager.cc:1397] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 03:01:33.392730 27377 tablet_bootstrap.cc:492] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: No bootstrap required, opened a new log
I20260928 03:01:33.392935 27378 tablet_bootstrap.cc:492] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: No bootstrap required, opened a new log
I20260928 03:01:33.392985 27377 ts_tablet_manager.cc:1397] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 03:01:33.393162 27378 ts_tablet_manager.cc:1397] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260928 03:01:33.400209 27378 raft_consensus.cc:348] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } } peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } }
I20260928 03:01:33.400537 27378 raft_consensus.cc:374] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:33.400667 27378 raft_consensus.cc:729] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7148b3104c42138425b9b052484de5, State: Initialized, Role: FOLLOWER
I20260928 03:01:33.400486 27376 raft_consensus.cc:348] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } } peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } }
I20260928 03:01:33.400903 27376 raft_consensus.cc:374] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:33.401032 27376 raft_consensus.cc:729] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db6101237d2491bb46fb86536f634d3, State: Initialized, Role: FOLLOWER
I20260928 03:01:33.401249 27378 consensus_queue.cc:260] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } } peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } }
I20260928 03:01:33.401585 27376 consensus_queue.cc:260] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } } peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } }
I20260928 03:01:33.402352 27358 heartbeater.cc:503] Master 127.12.182.126:37675 was elected leader, sending a full tablet report...
I20260928 03:01:33.402801 27378 ts_tablet_manager.cc:1428] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260928 03:01:33.402683 27377 raft_consensus.cc:348] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } } peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } }
I20260928 03:01:33.403076 27377 raft_consensus.cc:374] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:33.403185 27377 raft_consensus.cc:729] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99bc8617f7c4d118ebfa7fd224a9ca0, State: Initialized, Role: FOLLOWER
I20260928 03:01:33.403887 27377 consensus_queue.cc:260] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } } peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } }
I20260928 03:01:33.405469 27377 ts_tablet_manager.cc:1428] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260928 03:01:33.405969 27225 heartbeater.cc:503] Master 127.12.182.126:37675 was elected leader, sending a full tablet report...
I20260928 03:01:33.407225 27376 ts_tablet_manager.cc:1428] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
W20260928 03:01:33.538758 27359 tablet.cc:2369] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:01:33.572975 27226 tablet.cc:2369] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:01:33.592731 27093 tablet.cc:2369] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:01:33.730566 27383 raft_consensus.cc:482] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:01:33.730862 27383 raft_consensus.cc:504] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } } peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } }
I20260928 03:01:33.732530 27383 leader_election.cc:288] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513), f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375)
I20260928 03:01:33.738828 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.733060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:57593 (local address 127.12.182.67:42513)
0928 03:01:33.733235 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:33.733241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:33.733509 (+ 268us) server_negotiation.cc:407] Connection header received
0928 03:01:33.733834 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:33.733840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:33.733921 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:33.733976 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:33.734598 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.735380 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.737160 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.737959 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.738102 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.738177 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:33.738453 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 03:01:33.738573 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:01:33.738935 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.733056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:57751 (local address 127.12.182.66:37375)
0928 03:01:33.733243 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:33.733248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:33.734105 (+ 857us) server_negotiation.cc:407] Connection header received
0928 03:01:33.734279 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:33.734283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:33.734354 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:33.734424 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:33.734970 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.735644 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.737299 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.738104 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.738237 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.738306 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:33.738589 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 03:01:33.738707 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:01:33.738981 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.732782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:42513 (local address 127.12.182.65:57593)
0928 03:01:33.733404 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:33.733424 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:33.733586 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:33.734136 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:33.734139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:33.734161 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:33.734471 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.734479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.735504 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.735510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.737011 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.737024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.738151 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.738157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.738306 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.738330 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 03:01:33.738414 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 03:01:33.738531 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":1,"thread_start_us":306,"threads_started":1}
I20260928 03:01:33.739032 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.732803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:37375 (local address 127.12.182.65:57751)
0928 03:01:33.734008 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:33.734026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:33.734144 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:33.734466 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:33.734471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:33.734494 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:33.734841 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.734851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.735770 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.735776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.737163 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:33.737175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.738304 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.738310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:33.738457 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.738480 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 03:01:33.738554 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 03:01:33.738674 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"thread_start_us":890,"threads_started":1}
I20260928 03:01:33.739948 27313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf97262abde420781031b4d95eb57b9" candidate_uuid: "1db6101237d2491bb46fb86536f634d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7148b3104c42138425b9b052484de5" is_pre_election: true
I20260928 03:01:33.739953 27180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf97262abde420781031b4d95eb57b9" candidate_uuid: "1db6101237d2491bb46fb86536f634d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" is_pre_election: true
I20260928 03:01:33.740362 27313 raft_consensus.cc:2454] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1db6101237d2491bb46fb86536f634d3 in term 0.
I20260928 03:01:33.740378 27180 raft_consensus.cc:2454] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1db6101237d2491bb46fb86536f634d3 in term 0.
I20260928 03:01:33.740881 26980 leader_election.cc:302] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db6101237d2491bb46fb86536f634d3, 7a7148b3104c42138425b9b052484de5; no voters:
I20260928 03:01:33.741220 27383 raft_consensus.cc:2781] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:01:33.741362 27383 raft_consensus.cc:482] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:01:33.741458 27383 raft_consensus.cc:3037] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:33.743505 27383 raft_consensus.cc:504] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } } peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } }
I20260928 03:01:33.744074 27383 leader_election.cc:288] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 1 election: Requested vote from peers 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513), f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375)
I20260928 03:01:33.744434 27313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf97262abde420781031b4d95eb57b9" candidate_uuid: "1db6101237d2491bb46fb86536f634d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7148b3104c42138425b9b052484de5"
I20260928 03:01:33.744510 27180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf97262abde420781031b4d95eb57b9" candidate_uuid: "1db6101237d2491bb46fb86536f634d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0"
I20260928 03:01:33.744629 27313 raft_consensus.cc:3037] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:33.744678 27180 raft_consensus.cc:3037] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:33.747632 27180 raft_consensus.cc:2454] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1db6101237d2491bb46fb86536f634d3 in term 1.
I20260928 03:01:33.747632 27313 raft_consensus.cc:2454] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1db6101237d2491bb46fb86536f634d3 in term 1.
I20260928 03:01:33.747982 26980 leader_election.cc:302] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db6101237d2491bb46fb86536f634d3, 7a7148b3104c42138425b9b052484de5; no voters:
I20260928 03:01:33.748217 27383 raft_consensus.cc:2781] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:01:33.748611 27383 raft_consensus.cc:686] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 1 LEADER]: Becoming Leader. State: Replica: 1db6101237d2491bb46fb86536f634d3, State: Running, Role: LEADER
I20260928 03:01:33.749116 27383 consensus_queue.cc:237] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } } peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } }
I20260928 03:01:33.753274 26902 catalog_manager.cc:5495] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1db6101237d2491bb46fb86536f634d3 (127.12.182.65). New cstate: current_term: 1 leader_uuid: "1db6101237d2491bb46fb86536f634d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } health_report { overall_health: UNKNOWN } } }
I20260928 03:01:33.818617 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.813424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.12.182.65:44907)
0928 03:01:33.813654 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:33.813660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:33.813679 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:01:33.813844 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:33.813848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:33.813939 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:33.814049 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:01:33.814346 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.815096 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.816455 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.816853 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.816941 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.817411 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:01:33.818073 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:01:33.818198 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:33.818287 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 03:01:33.818367 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:01:33.827713 27180 raft_consensus.cc:1261] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 1 FOLLOWER]: Refusing update from remote peer 1db6101237d2491bb46fb86536f634d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:01:33.827711 27313 raft_consensus.cc:1261] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5 [term 1 FOLLOWER]: Refusing update from remote peer 1db6101237d2491bb46fb86536f634d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:01:33.828665 27387 consensus_queue.cc:1035] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:01:33.829084 27383 consensus_queue.cc:1035] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:01:33.840487 27390 mvcc.cc:205] Tried to move back new op lower bound from 7334152166707552256 to 7334152166399819776. Current Snapshot: MvccSnapshot[applied={T|T < 7334152166707552256}]
I20260928 03:01:33.854780 26903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:01:33.855070 26903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:01:33.858433 26903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Sending DeleteTablet for 3 replicas of tablet 9cf97262abde420781031b4d95eb57b9
I20260928 03:01:33.859501 27027 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf97262abde420781031b4d95eb57b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:33 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:54348
I20260928 03:01:33.859643 27160 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf97262abde420781031b4d95eb57b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:33 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:32824
I20260928 03:01:33.860370 27293 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf97262abde420781031b4d95eb57b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:33 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:32986
I20260928 03:01:33.859825 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 26964
I20260928 03:01:33.875500 27401 tablet_replica.cc:330] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: stopping tablet replica
I20260928 03:01:33.876078 27401 raft_consensus.cc:2229] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:01:33.876677 27401 raft_consensus.cc:2258] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:33.878770 27401 ts_tablet_manager.cc:1905] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:01:33.885872 27401 ts_tablet_manager.cc:1918] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:01:33.886087 27401 log.cc:1196] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/9cf97262abde420781031b4d95eb57b9
I20260928 03:01:33.886672 27401 ts_tablet_manager.cc:1939] T 9cf97262abde420781031b4d95eb57b9 P f99bc8617f7c4d118ebfa7fd224a9ca0: Deleting consensus metadata
I20260928 03:01:33.888068 27402 tablet_replica.cc:330] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: stopping tablet replica
I20260928 03:01:33.888473 26889 catalog_manager.cc:4841] TS f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375): tablet 9cf97262abde420781031b4d95eb57b9 (table test-table [id=3481e224f0e44581b3d70a5b61be9ce6]) successfully deleted
I20260928 03:01:33.888759 27402 raft_consensus.cc:2229] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:01:33.889318 27402 raft_consensus.cc:2258] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:33.891438 27402 ts_tablet_manager.cc:1905] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:01:33.893275 27400 tablet_replica.cc:330] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: stopping tablet replica
I20260928 03:01:33.893705 27400 raft_consensus.cc:2229] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:01:33.894356 27400 raft_consensus.cc:2258] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:33.896624 27400 ts_tablet_manager.cc:1905] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:01:33.897902 27402 ts_tablet_manager.cc:1918] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:01:33.898090 27402 log.cc:1196] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/9cf97262abde420781031b4d95eb57b9
I20260928 03:01:33.898629 27402 ts_tablet_manager.cc:1939] T 9cf97262abde420781031b4d95eb57b9 P 7a7148b3104c42138425b9b052484de5: Deleting consensus metadata
Sep 28 03:01:33 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 03:01:33.900319 26887 catalog_manager.cc:4841] TS 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513): tablet 9cf97262abde420781031b4d95eb57b9 (table test-table [id=3481e224f0e44581b3d70a5b61be9ce6]) successfully deleted
I20260928 03:01:33.903187 27400 ts_tablet_manager.cc:1918] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:01:33.903391 27400 log.cc:1196] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/9cf97262abde420781031b4d95eb57b9
I20260928 03:01:33.903815 27400 ts_tablet_manager.cc:1939] T 9cf97262abde420781031b4d95eb57b9 P 1db6101237d2491bb46fb86536f634d3: Deleting consensus metadata
I20260928 03:01:33.905122 26890 catalog_manager.cc:4841] TS 1db6101237d2491bb46fb86536f634d3 (127.12.182.65:44907): tablet 9cf97262abde420781031b4d95eb57b9 (table test-table [id=3481e224f0e44581b3d70a5b61be9ce6]) successfully deleted
I20260928 03:01:33.907227 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:33.862040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.12.182.65:44907)
0928 03:01:33.862213 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:33.862219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:33.885070 (+ 22851us) server_negotiation.cc:407] Connection header received
0928 03:01:33.885517 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:33.885523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:33.885628 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:33.885707 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:33.889739 (+ 4032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.890969 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.893753 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:33.894469 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:33.894570 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:33.900123 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:33.900137 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:33.900154 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:33.900200 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:33.904342 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:33.905224 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:33.905233 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:33.905236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:33.905404 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:33.905844 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:33.905848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:33.905851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:33.906218 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:33.906427 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:33.906786 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 03:01:33.906973 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 03:01:33.908048 27087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:34.005498 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 26964
I20260928 03:01:34.021099 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 27097
Sep 28 03:01:34 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 03:01:34.035607 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:34.021901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.12.182.66:37375)
0928 03:01:34.022081 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:34.022086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:34.022103 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:01:34.022223 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:34.022227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:34.022289 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:34.022379 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:34.023511 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:34.024242 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:34.025409 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:34.025868 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:34.025960 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:34.029979 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:34.029992 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:34.030008 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:34.030041 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:34.032528 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:34.033204 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:34.033234 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:34.033237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:34.033397 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:34.034045 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:34.034056 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:34.034059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:34.034503 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:34.034721 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:34.035082 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 03:01:34.035284 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:01:34.036213 27220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:34.123023 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 27097
I20260928 03:01:34.136586 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 27230
Sep 28 03:01:34 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 03:01:34.150300 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:34.137307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.12.182.67:42513)
0928 03:01:34.137484 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:34.137489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:34.137507 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:01:34.137615 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:34.137620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:34.137705 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:34.137784 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:34.139044 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:34.139755 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:34.140967 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:34.141348 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:34.141436 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:34.145488 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:34.145503 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:34.145519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:34.145551 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:34.148034 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:34.148607 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:34.148613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:34.148616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:34.148735 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:34.149002 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:34.149005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:34.149007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:34.149339 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:34.149571 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:34.149811 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 03:01:34.149984 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:01:34.150900 27353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:34.238570 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 27230
I20260928 03:01:34.251798 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 26871
I20260928 03:01:34.252915 26933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:34.359591 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 26871
I20260928 03:01:34.376334 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:37675
--webserver_interface=127.12.182.126
--webserver_port=32827
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:37675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:34.681450 27408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:34.681748 27408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:34.681887 27408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:34.688887 27408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:01:34.689009 27408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:34.689090 27408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:34.689168 27408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:01:34.689229 27408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:01:34.698709 27408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:37675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=32827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:34.699946 27408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:34.701166 27408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:34.721728 27414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:34.721675 27415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:34.721698 27417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:34.722447 27408 server_base.cc:1034] running on GCE node
I20260928 03:01:34.723284 27408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:34.724771 27408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:34.725972 27408 hybrid_clock.cc:648] HybridClock initialized: now 1790564494725906 us; error 50 us; skew 500 ppm
I20260928 03:01:34.726352 27408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:34 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564494, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:34.729557 27408 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:01:34.731102 27408 webserver.cc:460] Webserver started at http://127.12.182.126:32827/ using document root <none> and password file <none>
I20260928 03:01:34.731794 27408 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:34.731927 27408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:34.737586 27408 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:01:34.740356 27424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:34.741585 27408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 03:01:34.741802 27408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "c5993e2bd9b24067a036df645c9b3c55"
format_stamp: "Formatted at 2026-09-28 03:01:31 on dist-test-slave-k5rg"
server_key: "964eaaac5861ff4166696ffe8647e1ea"
server_key_iv: "05d8d3a8b57581a34f3126779ff23727"
server_key_version: "encryptionkey@0"
I20260928 03:01:34.742796 27408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:34.766400 27408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:34.770531 27408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:34.770793 27408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:34.792230 27408 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:37675
I20260928 03:01:34.792248 27475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:37675 every 8 connection(s)
I20260928 03:01:34.794114 27408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:01:34.794644 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 27408
I20260928 03:01:34.795691 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:44907
--local_ip_for_outbound_sockets=127.12.182.65
--tserver_master_addrs=127.12.182.126:37675
--webserver_port=40563
--webserver_interface=127.12.182.65
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:01:34.802075 27476 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:01:34.806562 27476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Bootstrap starting.
I20260928 03:01:34.834954 27476 log.cc:824] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:34.850155 27476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:01:34.850787 27476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Bootstrap complete.
I20260928 03:01:34.864483 27476 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:34.865643 27476 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5993e2bd9b24067a036df645c9b3c55, State: Initialized, Role: FOLLOWER
I20260928 03:01:34.866344 27476 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:34.866622 27476 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:01:34.866742 27476 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:01:34.866917 27476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 1 FOLLOWER]: Advancing to term 2
I20260928 03:01:34.872259 27476 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:34.872855 27476 leader_election.cc:302] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5993e2bd9b24067a036df645c9b3c55; no voters:
I20260928 03:01:34.873488 27476 leader_election.cc:288] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 03:01:34.873618 27480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 03:01:34.874794 27480 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 2 LEADER]: Becoming Leader. State: Replica: c5993e2bd9b24067a036df645c9b3c55, State: Running, Role: LEADER
I20260928 03:01:34.875294 27480 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:34.875682 27476 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:01:34.877555 27482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5993e2bd9b24067a036df645c9b3c55. Latest consensus state: current_term: 2 leader_uuid: "c5993e2bd9b24067a036df645c9b3c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } } }
I20260928 03:01:34.877573 27481 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c5993e2bd9b24067a036df645c9b3c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } } }
I20260928 03:01:34.877980 27481 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:34.878095 27482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:34.878856 27487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:01:34.889501 27487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:01:34.890930 27487 catalog_manager.cc:1259] Loaded cluster ID: 837f37c7c79b4744bea5d96e284d23d9
I20260928 03:01:34.891069 27487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:01:34.894109 27487 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 03:01:34.911413 27487 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 03:01:34.929796 27487 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:01:34.931628 27487 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Loaded TSK: 0
I20260928 03:01:34.933166 27487 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 03:01:35.170027 27478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:35.170320 27478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:35.170452 27478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:35.177937 27478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:35.178062 27478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:35.178206 27478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:01:35.188066 27478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:44907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=40563
--tserver_master_addrs=127.12.182.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:35.189361 27478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:35.190675 27478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:35.204862 27503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:35.204952 27504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:35.205583 27506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:35.205583 27478 server_base.cc:1034] running on GCE node
I20260928 03:01:35.206429 27478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:35.207634 27478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:35.208902 27478 hybrid_clock.cc:648] HybridClock initialized: now 1790564495208812 us; error 70 us; skew 500 ppm
I20260928 03:01:35.209290 27478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:35 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564495, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:35.213302 27478 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:01:35.215098 27478 webserver.cc:460] Webserver started at http://127.12.182.65:40563/ using document root <none> and password file <none>
I20260928 03:01:35.215976 27478 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:35.216131 27478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:35.223212 27478 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:01:35.226512 27513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:35.227991 27478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:01:35.228147 27478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "1db6101237d2491bb46fb86536f634d3"
format_stamp: "Formatted at 2026-09-28 03:01:32 on dist-test-slave-k5rg"
server_key: "e09f2ee71aed04288040663e956489fc"
server_key_iv: "a83c60281eee02ea1f73c5b6475cc138"
server_key_version: "encryptionkey@0"
I20260928 03:01:35.229393 27478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:35.246873 27478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:35.252051 27478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:35.252329 27478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:35.253407 27478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:35.254993 27478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:35.255097 27478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:35.255192 27478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:35.255259 27478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:35.288160 27478 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:44907
I20260928 03:01:35.288226 27625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:44907 every 8 connection(s)
I20260928 03:01:35.289522 27478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:01:35.290895 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 27478
I20260928 03:01:35.291540 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:37375
--local_ip_for_outbound_sockets=127.12.182.66
--tserver_master_addrs=127.12.182.126:37675
--webserver_port=44137
--webserver_interface=127.12.182.66
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:01:35 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564495, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:35.316197 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:35.292600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:55593 (local address 127.12.182.126:37675)
0928 03:01:35.298449 (+ 5849us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:35.298459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:35.298498 (+ 39us) server_negotiation.cc:407] Connection header received
0928 03:01:35.298794 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:35.298807 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:35.299123 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:35.299437 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:35.301185 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:35.302443 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:35.304548 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:35.305161 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:35.305288 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:35.309037 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:35.309044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:35.309058 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:35.309098 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:35.311795 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:35.312494 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:35.312500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:35.312505 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:35.312661 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:35.313104 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:35.313109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:35.313112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:35.313653 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:35.313884 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:35.314612 (+ 728us) server_negotiation.cc:299] Negotiation successful
0928 03:01:35.315010 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5664,"thread_start_us":5543,"threads_started":1}
I20260928 03:01:35.316197 27629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:35.292716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:55593)
0928 03:01:35.293568 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:35.293643 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:35.295823 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:35.299603 (+ 3780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:35.299614 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:35.300284 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:35.300952 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:35.300975 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:35.302595 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:35.302601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:35.304400 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:35.304422 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:35.305377 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:35.305383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:35.305664 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:35.306327 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:35.306352 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:35.308782 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:35.311946 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:35.311955 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:35.311969 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:35.312370 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:35.312811 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:35.312816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:35.312820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:35.312996 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:35.313791 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:35.313798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:35.314129 (+ 331us) client_negotiation.cc:769] Sending connection context
0928 03:01:35.314392 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 03:01:35.314929 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":318,"threads_started":1}
I20260928 03:01:35.319264 27626 heartbeater.cc:347] Connected to a master server at 127.12.182.126:37675
I20260928 03:01:35.319536 27626 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:35.320214 27626 heartbeater.cc:511] Master 127.12.182.126:37675 requested a full tablet report, sending...
I20260928 03:01:35.322424 27441 ts_manager.cc:194] Registered new tserver with Master: 1db6101237d2491bb46fb86536f634d3 (127.12.182.65:44907)
I20260928 03:01:35.324501 27441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:55593
W20260928 03:01:35.603799 27630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:35.604040 27630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:35.604137 27630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:35.611004 27630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:35.611102 27630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:35.611217 27630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:01:35.620379 27630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:37375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=44137
--tserver_master_addrs=127.12.182.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:35.621539 27630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:35.622807 27630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:35.634851 27636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:35.634944 27635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:35.634910 27638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:35.635614 27630 server_base.cc:1034] running on GCE node
I20260928 03:01:35.636155 27630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:35.636974 27630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:35.638159 27630 hybrid_clock.cc:648] HybridClock initialized: now 1790564495638115 us; error 31 us; skew 500 ppm
I20260928 03:01:35.638434 27630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:35 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564495, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:35.641183 27630 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:01:35.642421 27630 webserver.cc:460] Webserver started at http://127.12.182.66:44137/ using document root <none> and password file <none>
I20260928 03:01:35.642994 27630 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:35.643095 27630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:35.647936 27630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 03:01:35.650315 27645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:35.651475 27630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 03:01:35.651609 27630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0"
format_stamp: "Formatted at 2026-09-28 03:01:32 on dist-test-slave-k5rg"
server_key: "1e02d12110499b268f32676e45693079"
server_key_iv: "a102abdd031747b469e2ff7c89ab46e6"
server_key_version: "encryptionkey@0"
I20260928 03:01:35.652477 27630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:35.673884 27630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:35.677484 27630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:35.677729 27630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:35.678606 27630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:35.679800 27630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:35.679901 27630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:35.680001 27630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:35.680068 27630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:35.708514 27630 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:37375
I20260928 03:01:35.708577 27757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:37375 every 8 connection(s)
I20260928 03:01:35.709867 27630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:01:35.715476 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 27630
I20260928 03:01:35.716203 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:42513
--local_ip_for_outbound_sockets=127.12.182.67
--tserver_master_addrs=127.12.182.126:37675
--webserver_port=32865
--webserver_interface=127.12.182.67
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:01:35 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564495, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:35.729466 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:35.712165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:34199 (local address 127.12.182.126:37675)
0928 03:01:35.712365 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:35.712371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:35.713412 (+ 1041us) server_negotiation.cc:407] Connection header received
0928 03:01:35.715305 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:35.715309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:35.715383 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:35.715462 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:35.717316 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:35.718134 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:35.720143 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:35.720564 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:35.720657 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:35.724272 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:35.724279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:35.724283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:35.724316 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:35.726691 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:35.727325 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:35.727331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:35.727335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:35.727462 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:35.727847 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:35.727851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:35.727855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:35.728113 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:35.728244 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:35.728897 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 03:01:35.729120 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:01:35.730417 27760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:35.712606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:34199)
0928 03:01:35.713273 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:35.713308 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:35.715012 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:35.715726 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:35.715736 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:35.716419 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:35.717126 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:35.717149 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:35.718292 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:35.718297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:35.719993 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:35.720006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:35.720790 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:35.720796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:35.721048 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:35.721691 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:35.721722 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:35.724058 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:35.726816 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:35.726825 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:35.726841 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:35.727215 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:35.727562 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:35.727566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:35.727570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:35.727739 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:35.728231 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:35.728238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:35.728544 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 03:01:35.728774 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 03:01:35.729249 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":253,"threads_started":1}
I20260928 03:01:35.731804 27758 heartbeater.cc:347] Connected to a master server at 127.12.182.126:37675
I20260928 03:01:35.732089 27758 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:35.732772 27758 heartbeater.cc:511] Master 127.12.182.126:37675 requested a full tablet report, sending...
I20260928 03:01:35.734148 27441 ts_manager.cc:194] Registered new tserver with Master: f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375)
I20260928 03:01:35.734970 27441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:34199
W20260928 03:01:36.026978 27761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:36.027248 27761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:36.027388 27761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:36.034606 27761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:36.034751 27761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:36.035000 27761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:01:36.044610 27761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:42513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=32865
--tserver_master_addrs=127.12.182.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:36.045827 27761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:36.047098 27761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:36.059254 27769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:36.059357 27766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:36.059223 27767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:36.060505 27761 server_base.cc:1034] running on GCE node
I20260928 03:01:36.061082 27761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:36.062005 27761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:36.063210 27761 hybrid_clock.cc:648] HybridClock initialized: now 1790564496063159 us; error 41 us; skew 500 ppm
I20260928 03:01:36.063504 27761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:36 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564496, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:36.066578 27761 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:01:36.067991 27761 webserver.cc:460] Webserver started at http://127.12.182.67:32865/ using document root <none> and password file <none>
I20260928 03:01:36.068645 27761 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:36.068778 27761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:36.074031 27761 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:01:36.076624 27776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:36.077967 27761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:01:36.078106 27761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "7a7148b3104c42138425b9b052484de5"
format_stamp: "Formatted at 2026-09-28 03:01:33 on dist-test-slave-k5rg"
server_key: "5709d05926ca05aee62759d615355b78"
server_key_iv: "7217f16bef8198b9a2e0a3d230826a27"
server_key_version: "encryptionkey@0"
I20260928 03:01:36.079027 27761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:36.103209 27761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:36.106932 27761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:36.107168 27761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:36.108181 27761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:36.109493 27761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:36.109591 27761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:36.109720 27761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:36.109793 27761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:36.138929 27761 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:42513
I20260928 03:01:36.138993 27888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:42513 every 8 connection(s)
I20260928 03:01:36.140304 27761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:01:36.148636 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 27761
Sep 28 03:01:36 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564496, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:36.160457 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.142739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:47193 (local address 127.12.182.126:37675)
0928 03:01:36.142955 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.142962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.144074 (+ 1112us) server_negotiation.cc:407] Connection header received
0928 03:01:36.145825 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.145831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.145916 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.146004 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:36.147401 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.148122 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.150268 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.150850 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.150966 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.154446 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:36.154451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:36.154454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:36.154483 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:36.157335 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:36.158021 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:36.158025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:36.158027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:36.158160 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:36.158573 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:36.158577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:36.158579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:36.158958 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:36.159092 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.159862 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.160106 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 03:01:36.161065 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.143230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:47193)
0928 03:01:36.143929 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.143965 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:36.145554 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:36.146192 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:36.146200 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:36.146672 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:36.147248 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.147267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.148284 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.148290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.150095 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.150112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.151106 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.151111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.151316 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.151949 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:36.151970 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:36.154258 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:36.157456 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:36.157464 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:36.157475 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:36.157907 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:36.158271 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:36.158276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:36.158281 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:36.158458 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:36.159077 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:36.159084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:36.159389 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 03:01:36.159675 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 03:01:36.160030 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":258,"threads_started":1}
I20260928 03:01:36.162532 27889 heartbeater.cc:347] Connected to a master server at 127.12.182.126:37675
I20260928 03:01:36.162818 27889 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:36.163648 27889 heartbeater.cc:511] Master 127.12.182.126:37675 requested a full tablet report, sending...
I20260928 03:01:36.164101 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.149518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.12.182.126:37675)
0928 03:01:36.150221 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.150226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.150541 (+ 315us) server_negotiation.cc:407] Connection header received
0928 03:01:36.150689 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.150692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.150745 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.150834 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:36.152432 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.153379 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.154790 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.155353 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.155448 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.158119 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:36.158125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:36.158131 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:36.158161 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:36.161133 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:36.161990 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:36.161993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:36.161995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:36.162135 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:36.162592 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:36.162596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:36.162598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:36.162878 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:36.163093 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.163453 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.163615 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":446,"threads_started":1}
I20260928 03:01:36.165205 27441 ts_manager.cc:194] Registered new tserver with Master: 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513)
I20260928 03:01:36.166160 27441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:47193
I20260928 03:01:36.166783 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:01:36.172963 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.167609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.12.182.126:37675)
0928 03:01:36.167745 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.167750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.168196 (+ 446us) server_negotiation.cc:407] Connection header received
0928 03:01:36.168406 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.168410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.168523 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.168635 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:01:36.169034 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.169744 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.171088 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.171496 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.171598 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.172053 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:01:36.172443 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:01:36.172567 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.172665 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.172738 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 03:01:36.173743 27441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:01:36.180958 27441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:01:36.218164 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.192524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.12.182.65:44907)
0928 03:01:36.211089 (+ 18565us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.211098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.211124 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:01:36.211217 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.211225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.211437 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.211617 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:36.212367 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.213568 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.215552 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.216989 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.217128 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.217345 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.217702 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.217871 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18396,"thread_start_us":19268,"threads_started":1}
I20260928 03:01:36.218204 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.192007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.0.0.1:54366)
0928 03:01:36.207174 (+ 15167us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.207194 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:36.207318 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:36.211835 (+ 4517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:36.211840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:36.211865 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:36.212165 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.212175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.213699 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.213704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.215403 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.215415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.217185 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.217191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.217392 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.217449 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 03:01:36.217568 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 03:01:36.217777 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15023,"mutex_wait_us":20,"thread_start_us":12438,"threads_started":1}
I20260928 03:01:36.218513 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.191862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:37375 (local address 127.0.0.1:32846)
0928 03:01:36.204854 (+ 12992us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.204907 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:36.205073 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:36.210501 (+ 5428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:36.210508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:36.210537 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:36.210922 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.210936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.212771 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.212777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.215590 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.215600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.217767 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.217773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.217971 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.218001 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 03:01:36.218097 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 03:01:36.218219 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12717,"mutex_wait_us":143,"thread_start_us":15532,"threads_started":1}
I20260928 03:01:36.219355 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.192219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.12.182.66:37375)
0928 03:01:36.210010 (+ 17791us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.210018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.210044 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:01:36.210115 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.210120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.210301 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.210471 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:36.211175 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.212655 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.215735 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.217091 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.217193 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.217380 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.218873 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.219032 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17636,"thread_start_us":17507,"threads_started":1}
I20260928 03:01:36.219770 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.192830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.12.182.67:42513)
0928 03:01:36.209011 (+ 16181us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.209023 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.209941 (+ 918us) server_negotiation.cc:407] Connection header received
0928 03:01:36.210058 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.210068 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.210340 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.210523 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:36.211487 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.212832 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.217455 (+ 4623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.218762 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.218912 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.219125 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.219308 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.219458 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15985,"thread_start_us":16183,"threads_started":1}
I20260928 03:01:36.219856 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.192522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:42513 (local address 127.0.0.1:32994)
0928 03:01:36.209662 (+ 17140us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.209681 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:36.209815 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:36.210830 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:36.210835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:36.210862 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:36.211290 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.211303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.212966 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.212970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.215575 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.215589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.218977 (+ 3388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.218983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.219130 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.219160 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 03:01:36.219246 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 03:01:36.219370 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16999,"mutex_wait_us":59,"thread_start_us":16863,"threads_started":1}
I20260928 03:01:36.222245 27561 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9f8a68a8f0410492ebeb79a4d56f64 (DEFAULT_TABLE table=test-table [id=ca8fed10ce45484bb55ac57a0fb42fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:36.222721 27693 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9f8a68a8f0410492ebeb79a4d56f64 (DEFAULT_TABLE table=test-table [id=ca8fed10ce45484bb55ac57a0fb42fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:36.223471 27561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9f8a68a8f0410492ebeb79a4d56f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:36.223176 27824 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9f8a68a8f0410492ebeb79a4d56f64 (DEFAULT_TABLE table=test-table [id=ca8fed10ce45484bb55ac57a0fb42fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:36.224092 27693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9f8a68a8f0410492ebeb79a4d56f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:36.224403 27824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9f8a68a8f0410492ebeb79a4d56f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:36.235536 27905 tablet_bootstrap.cc:492] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Bootstrap starting.
I20260928 03:01:36.238456 27906 tablet_bootstrap.cc:492] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: Bootstrap starting.
I20260928 03:01:36.238698 27905 tablet_bootstrap.cc:654] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:36.240033 27905 log.cc:824] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:36.241655 27906 tablet_bootstrap.cc:654] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:36.242905 27906 log.cc:824] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:36.243157 27905 tablet_bootstrap.cc:492] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: No bootstrap required, opened a new log
I20260928 03:01:36.243436 27905 ts_tablet_manager.cc:1397] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 03:01:36.243651 27907 tablet_bootstrap.cc:492] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: Bootstrap starting.
I20260928 03:01:36.247051 27906 tablet_bootstrap.cc:492] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: No bootstrap required, opened a new log
I20260928 03:01:36.247059 27907 tablet_bootstrap.cc:654] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:36.247280 27906 ts_tablet_manager.cc:1397] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 03:01:36.248200 27907 log.cc:824] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:36.251019 27907 tablet_bootstrap.cc:492] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: No bootstrap required, opened a new log
I20260928 03:01:36.251253 27907 ts_tablet_manager.cc:1397] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 03:01:36.253017 27905 raft_consensus.cc:348] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:36.253432 27905 raft_consensus.cc:374] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:36.253530 27905 raft_consensus.cc:729] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db6101237d2491bb46fb86536f634d3, State: Initialized, Role: FOLLOWER
I20260928 03:01:36.254184 27905 consensus_queue.cc:260] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:36.254253 27906 raft_consensus.cc:348] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:36.254555 27906 raft_consensus.cc:374] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:36.254644 27906 raft_consensus.cc:729] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99bc8617f7c4d118ebfa7fd224a9ca0, State: Initialized, Role: FOLLOWER
I20260928 03:01:36.255172 27906 consensus_queue.cc:260] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:36.256253 27758 heartbeater.cc:503] Master 127.12.182.126:37675 was elected leader, sending a full tablet report...
I20260928 03:01:36.257395 27906 ts_tablet_manager.cc:1428] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260928 03:01:36.258252 27626 heartbeater.cc:503] Master 127.12.182.126:37675 was elected leader, sending a full tablet report...
I20260928 03:01:36.258842 27905 ts_tablet_manager.cc:1428] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260928 03:01:36.264106 27907 raft_consensus.cc:348] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:36.264444 27907 raft_consensus.cc:374] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:36.264535 27907 raft_consensus.cc:729] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7148b3104c42138425b9b052484de5, State: Initialized, Role: FOLLOWER
I20260928 03:01:36.265044 27907 consensus_queue.cc:260] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:36.265967 27889 heartbeater.cc:503] Master 127.12.182.126:37675 was elected leader, sending a full tablet report...
I20260928 03:01:36.266541 27907 ts_tablet_manager.cc:1428] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260928 03:01:36.291426 27627 tablet.cc:2369] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:01:36.328550 27913 raft_consensus.cc:482] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:01:36.328819 27913 raft_consensus.cc:504] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:36.330507 27913 leader_election.cc:288] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375), 1db6101237d2491bb46fb86536f634d3 (127.12.182.65:44907)
I20260928 03:01:36.336053 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.331046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:45671 (local address 127.12.182.66:37375)
0928 03:01:36.331243 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.331250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.331273 (+ 23us) server_negotiation.cc:407] Connection header received
0928 03:01:36.331460 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.331464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.331527 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.331605 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:36.332022 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.332694 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.334407 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.335222 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.335387 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.335460 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.335696 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.335814 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:01:36.336059 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.330975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:37375 (local address 127.12.182.67:45671)
0928 03:01:36.331147 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.331167 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:36.331294 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:36.331651 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:36.331656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:36.331692 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:36.331874 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.331884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.332815 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.332819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.334254 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.334267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.335384 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.335390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.335557 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.335581 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 03:01:36.335660 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 03:01:36.335786 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":2}
I20260928 03:01:36.336964 27713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9f8a68a8f0410492ebeb79a4d56f64" candidate_uuid: "7a7148b3104c42138425b9b052484de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" is_pre_election: true
I20260928 03:01:36.337131 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.331071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:32939 (local address 127.12.182.65:44907)
0928 03:01:36.331259 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.331264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.332021 (+ 757us) server_negotiation.cc:407] Connection header received
0928 03:01:36.332165 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.332168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.332224 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.332304 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:36.332811 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.333502 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.335252 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.336102 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.336256 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.336318 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.336773 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.336891 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:01:36.337208 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.330975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.12.182.67:32939)
0928 03:01:36.331918 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.331932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:36.332038 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:36.332333 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:36.332337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:36.332359 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:36.332659 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.332668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.333660 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.333663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.335103 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:36.335115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.336437 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.336443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:36.336602 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.336635 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 03:01:36.336724 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 03:01:36.336842 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":243,"thread_start_us":376,"threads_started":1}
I20260928 03:01:36.337448 27713 raft_consensus.cc:2454] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a7148b3104c42138425b9b052484de5 in term 0.
I20260928 03:01:36.338148 27779 leader_election.cc:302] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a7148b3104c42138425b9b052484de5, f99bc8617f7c4d118ebfa7fd224a9ca0; no voters:
I20260928 03:01:36.338145 27581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9f8a68a8f0410492ebeb79a4d56f64" candidate_uuid: "7a7148b3104c42138425b9b052484de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db6101237d2491bb46fb86536f634d3" is_pre_election: true
I20260928 03:01:36.338492 27913 raft_consensus.cc:2781] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:01:36.338603 27913 raft_consensus.cc:482] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:01:36.338690 27913 raft_consensus.cc:3037] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:36.338591 27581 raft_consensus.cc:2454] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a7148b3104c42138425b9b052484de5 in term 0.
I20260928 03:01:36.340834 27913 raft_consensus.cc:504] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:36.341428 27913 leader_election.cc:288] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [CANDIDATE]: Term 1 election: Requested vote from peers f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375), 1db6101237d2491bb46fb86536f634d3 (127.12.182.65:44907)
I20260928 03:01:36.341876 27713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9f8a68a8f0410492ebeb79a4d56f64" candidate_uuid: "7a7148b3104c42138425b9b052484de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0"
I20260928 03:01:36.341894 27581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9f8a68a8f0410492ebeb79a4d56f64" candidate_uuid: "7a7148b3104c42138425b9b052484de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db6101237d2491bb46fb86536f634d3"
I20260928 03:01:36.342063 27581 raft_consensus.cc:3037] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:36.342056 27713 raft_consensus.cc:3037] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:36.344137 27581 raft_consensus.cc:2454] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a7148b3104c42138425b9b052484de5 in term 1.
I20260928 03:01:36.344236 27713 raft_consensus.cc:2454] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a7148b3104c42138425b9b052484de5 in term 1.
I20260928 03:01:36.344517 27780 leader_election.cc:302] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db6101237d2491bb46fb86536f634d3, 7a7148b3104c42138425b9b052484de5; no voters:
I20260928 03:01:36.344810 27913 raft_consensus.cc:2781] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:01:36.345268 27913 raft_consensus.cc:686] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 1 LEADER]: Becoming Leader. State: Replica: 7a7148b3104c42138425b9b052484de5, State: Running, Role: LEADER
I20260928 03:01:36.345832 27913 consensus_queue.cc:237] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:36.350344 27440 catalog_manager.cc:5495] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a7148b3104c42138425b9b052484de5 (127.12.182.67). New cstate: current_term: 1 leader_uuid: "7a7148b3104c42138425b9b052484de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } health_report { overall_health: HEALTHY } } }
W20260928 03:01:36.391757 27890 tablet.cc:2369] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:01:36.394054 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.389304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.12.182.67:42513)
0928 03:01:36.389473 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.389478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.389494 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:01:36.389652 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.389656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.389742 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.389859 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:01:36.390167 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.390875 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.392191 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.392676 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.392767 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.393152 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:01:36.393641 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:01:36.393744 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.393804 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.393866 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:01:36.401734 27581 raft_consensus.cc:1261] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Refusing update from remote peer 7a7148b3104c42138425b9b052484de5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:01:36.401726 27713 raft_consensus.cc:1261] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 1 FOLLOWER]: Refusing update from remote peer 7a7148b3104c42138425b9b052484de5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:01:36.402730 27915 consensus_queue.cc:1035] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:01:36.403049 27913 consensus_queue.cc:1035] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:01:36.412972 27918 mvcc.cc:205] Tried to move back new op lower bound from 7334152177250684928 to 7334152177036398592. Current Snapshot: MvccSnapshot[applied={T|T < 7334152177250684928}]
I20260928 03:01:36.414328 27920 mvcc.cc:205] Tried to move back new op lower bound from 7334152177250684928 to 7334152177036398592. Current Snapshot: MvccSnapshot[applied={T|T < 7334152177250684928}]
I20260928 03:01:36.415903 27919 mvcc.cc:205] Tried to move back new op lower bound from 7334152177250684928 to 7334152177036398592. Current Snapshot: MvccSnapshot[applied={T|T < 7334152177250684928}]
I20260928 03:01:36.427949 27440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:01:36.428217 27440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:01:36.431607 27440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Sending DeleteTablet for 3 replicas of tablet 3d9f8a68a8f0410492ebeb79a4d56f64
I20260928 03:01:36.432686 27561 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d9f8a68a8f0410492ebeb79a4d56f64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:36 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:54366
I20260928 03:01:36.432744 27693 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d9f8a68a8f0410492ebeb79a4d56f64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:36 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:32846
I20260928 03:01:36.432874 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 27478
I20260928 03:01:36.433586 27824 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d9f8a68a8f0410492ebeb79a4d56f64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:36 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:32994
I20260928 03:01:36.450431 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.433586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.12.182.65:44907)
0928 03:01:36.433785 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.433791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.433966 (+ 175us) server_negotiation.cc:407] Connection header received
0928 03:01:36.434082 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.434088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.434170 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.434292 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:36.436034 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.437268 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.439198 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.439871 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.439980 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.443242 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:36.443256 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:36.443275 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:36.443313 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:36.447443 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:36.448199 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:36.448208 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:36.448212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:36.448372 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:36.448717 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:36.448722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:36.448725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:36.449201 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:36.449493 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.449803 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.450028 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:01:36.451327 27621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:36.458446 27930 tablet_replica.cc:330] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: stopping tablet replica
I20260928 03:01:36.458904 27930 raft_consensus.cc:2229] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:01:36.459666 27930 raft_consensus.cc:2258] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:36.463799 27930 ts_tablet_manager.cc:1905] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:01:36.464879 27929 tablet_replica.cc:330] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: stopping tablet replica
I20260928 03:01:36.465332 27929 raft_consensus.cc:2229] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:01:36.465938 27929 raft_consensus.cc:2258] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:36.467878 27929 ts_tablet_manager.cc:1905] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:01:36.472062 27930 ts_tablet_manager.cc:1918] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:01:36.472210 27930 log.cc:1196] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/3d9f8a68a8f0410492ebeb79a4d56f64
I20260928 03:01:36.472679 27930 ts_tablet_manager.cc:1939] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 7a7148b3104c42138425b9b052484de5: Deleting consensus metadata
I20260928 03:01:36.474375 27425 catalog_manager.cc:4841] TS 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513): tablet 3d9f8a68a8f0410492ebeb79a4d56f64 (table test-table [id=ca8fed10ce45484bb55ac57a0fb42fd6]) successfully deleted
I20260928 03:01:36.478097 27929 ts_tablet_manager.cc:1918] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:01:36.478291 27929 log.cc:1196] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/3d9f8a68a8f0410492ebeb79a4d56f64
I20260928 03:01:36.479004 27929 ts_tablet_manager.cc:1939] T 3d9f8a68a8f0410492ebeb79a4d56f64 P f99bc8617f7c4d118ebfa7fd224a9ca0: Deleting consensus metadata
I20260928 03:01:36.480786 27427 catalog_manager.cc:4841] TS f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375): tablet 3d9f8a68a8f0410492ebeb79a4d56f64 (table test-table [id=ca8fed10ce45484bb55ac57a0fb42fd6]) successfully deleted
I20260928 03:01:36.562866 27928 tablet_replica.cc:330] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: stopping tablet replica
I20260928 03:01:36.563159 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 27478
I20260928 03:01:36.578291 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 27630
W20260928 03:01:36.578373 27428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.182.65:44907 (error 108)
I20260928 03:01:36.580345 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.579877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.0.0.1:54390)
0928 03:01:36.580099 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.580208 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 03:01:36.580576 27428 catalog_manager.cc:4584] TS 1db6101237d2491bb46fb86536f634d3 (127.12.182.65:44907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d9f8a68a8f0410492ebeb79a4d56f64: Network error: Client connection negotiation failed: client connection to 127.12.182.65:44907: connect: Connection refused (error 111)
I20260928 03:01:36.594468 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.579606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.12.182.66:37375)
0928 03:01:36.579772 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.579776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.579792 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:01:36.579844 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.579848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.579907 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.580001 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:36.581875 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.582944 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.584381 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.584963 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.585094 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.587846 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:36.587864 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:36.587884 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:36.587941 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:36.591663 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:36.592318 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:36.592327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:36.592331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:36.592483 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:36.592791 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:36.592795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:36.592799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:36.593235 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:36.593531 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.593868 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.594089 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 03:01:36.595240 27753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:36.634254 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.633713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.0.0.1:54394)
0928 03:01:36.633945 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.634052 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 03:01:36.635422 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.635028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.0.0.1:54408)
0928 03:01:36.635175 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.635283 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 03:01:36.681588 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 27630
I20260928 03:01:36.695366 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 27761
I20260928 03:01:36.696094 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.695582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.0.0.1:54414)
0928 03:01:36.695798 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.695896 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 03:01:36.697228 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.696834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.0.0.1:54428)
0928 03:01:36.696999 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.697085 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 03:01:36.712699 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.700206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.12.182.67:42513)
0928 03:01:36.700386 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:36.700392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:36.700413 (+ 21us) server_negotiation.cc:407] Connection header received
0928 03:01:36.700629 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:36.700633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:36.700695 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:36.700785 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:36.702186 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.702990 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.704219 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:36.704682 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:36.704790 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:36.707186 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:36.707199 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:36.707213 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:36.707253 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:36.710074 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:36.710745 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:36.710751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:36.710754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:36.710899 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:36.711241 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:36.711244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:36.711246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:36.711587 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:36.711862 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:36.712164 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 03:01:36.712377 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 03:01:36.713431 27884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:36.777814 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.777380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.0.0.1:54430)
0928 03:01:36.777553 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.777665 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 03:01:36.778775 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.778418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.0.0.1:54438)
0928 03:01:36.778598 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.778667 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 03:01:36.834609 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 27761
I20260928 03:01:36.850489 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 27408
I20260928 03:01:36.851557 27471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:36.960958 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 27408
I20260928 03:01:36.961035 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:36.960571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44907 (local address 127.0.0.1:54442)
0928 03:01:36.960765 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:36.960863 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 03:01:36.978367 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:37675
--webserver_interface=127.12.182.126
--webserver_port=32827
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:37675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:37.281042 27936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:37.281291 27936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:37.281416 27936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:37.288806 27936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:01:37.288900 27936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:37.288965 27936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:37.289042 27936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:01:37.289098 27936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:01:37.298434 27936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:37675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=32827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:37.299604 27936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:37.300804 27936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:37.311430 27941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:37.311432 27942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:37.311370 27944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:37.312947 27936 server_base.cc:1034] running on GCE node
I20260928 03:01:37.313681 27936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:37.315023 27936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:37.316224 27936 hybrid_clock.cc:648] HybridClock initialized: now 1790564497316180 us; error 39 us; skew 500 ppm
I20260928 03:01:37.316532 27936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:37 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564497, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:37.323493 27936 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:01:37.324893 27936 webserver.cc:460] Webserver started at http://127.12.182.126:32827/ using document root <none> and password file <none>
I20260928 03:01:37.325517 27936 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:37.325646 27936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:37.330592 27936 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 03:01:37.333158 27951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:37.334383 27936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:01:37.334532 27936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "c5993e2bd9b24067a036df645c9b3c55"
format_stamp: "Formatted at 2026-09-28 03:01:31 on dist-test-slave-k5rg"
server_key: "964eaaac5861ff4166696ffe8647e1ea"
server_key_iv: "05d8d3a8b57581a34f3126779ff23727"
server_key_version: "encryptionkey@0"
I20260928 03:01:37.335414 27936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:37.385062 27936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:37.388715 27936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:37.388927 27936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:37.405829 27936 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:37675
I20260928 03:01:37.405864 28002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:37675 every 8 connection(s)
I20260928 03:01:37.407166 27936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:01:37.413654 28003 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:01:37.416692 28003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Bootstrap starting.
I20260928 03:01:37.417531 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 27936
I20260928 03:01:37.418536 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:44907
--local_ip_for_outbound_sockets=127.12.182.65
--tserver_master_addrs=127.12.182.126:37675
--webserver_port=40563
--webserver_interface=127.12.182.65
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:01:37.438267 28003 log.cc:824] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:37.453855 28003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:01:37.454312 28003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55: Bootstrap complete.
I20260928 03:01:37.462903 28003 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:37.463706 28003 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5993e2bd9b24067a036df645c9b3c55, State: Initialized, Role: FOLLOWER
I20260928 03:01:37.464278 28003 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:37.464484 28003 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:01:37.464594 28003 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:01:37.464749 28003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 2 FOLLOWER]: Advancing to term 3
I20260928 03:01:37.468492 28003 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:37.468978 28003 leader_election.cc:302] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5993e2bd9b24067a036df645c9b3c55; no voters:
I20260928 03:01:37.469568 28003 leader_election.cc:288] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 03:01:37.469727 28007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 03:01:37.470752 28007 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [term 3 LEADER]: Becoming Leader. State: Replica: c5993e2bd9b24067a036df645c9b3c55, State: Running, Role: LEADER
I20260928 03:01:37.471268 28007 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } }
I20260928 03:01:37.471676 28003 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:01:37.473742 28008 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c5993e2bd9b24067a036df645c9b3c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } } }
I20260928 03:01:37.474128 28008 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:37.474581 28009 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5993e2bd9b24067a036df645c9b3c55. Latest consensus state: current_term: 3 leader_uuid: "c5993e2bd9b24067a036df645c9b3c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5993e2bd9b24067a036df645c9b3c55" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37675 } } }
I20260928 03:01:37.474884 28009 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5993e2bd9b24067a036df645c9b3c55 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:37.475078 28015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:01:37.499234 28015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:01:37.500801 28015 catalog_manager.cc:1259] Loaded cluster ID: 837f37c7c79b4744bea5d96e284d23d9
I20260928 03:01:37.500926 28015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 03:01:37.503089 28015 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc7e48c09ad google::LogMessage::Fail() at ??:0
@ 0x7fc7e48c388a google::LogMessage::SendToLog() at ??:0
@ 0x7fc7e48c04de google::LogMessage::Flush() at ??:0
@ 0x7fc7e48c1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc7fa2f3512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc7fa2f0976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc7fa378610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc7fa378439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc7e750c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc7e750f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc7e750f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc7e74eaeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc7ea7c46db start_thread at ??:0
@ 0x7fc7e375a61f clone at ??:0
W20260928 03:01:37.787207 28005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:37.787470 28005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:37.787587 28005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:37.794925 28005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:37.795049 28005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:37.795219 28005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:01:37.807292 28005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:44907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=40563
--tserver_master_addrs=127.12.182.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:37.808454 28005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:37.809728 28005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:37.822101 28047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:37.822166 28044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:37.822187 28045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:37.823027 28005 server_base.cc:1034] running on GCE node
I20260928 03:01:37.823560 28005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:37.824397 28005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:37.825587 28005 hybrid_clock.cc:648] HybridClock initialized: now 1790564497825544 us; error 31 us; skew 500 ppm
I20260928 03:01:37.825927 28005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:37 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564497, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:37.828792 28005 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:01:37.830171 28005 webserver.cc:460] Webserver started at http://127.12.182.65:40563/ using document root <none> and password file <none>
I20260928 03:01:37.830802 28005 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:37.830919 28005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:37.835867 28005 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 03:01:37.838487 28054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:37.839718 28005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:01:37.839854 28005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "1db6101237d2491bb46fb86536f634d3"
format_stamp: "Formatted at 2026-09-28 03:01:32 on dist-test-slave-k5rg"
server_key: "e09f2ee71aed04288040663e956489fc"
server_key_iv: "a83c60281eee02ea1f73c5b6475cc138"
server_key_version: "encryptionkey@0"
I20260928 03:01:37.840732 28005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:37.864670 28005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:37.868673 28005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:37.868909 28005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:37.869879 28005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:37.871556 28061 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 03:01:37.876792 28005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 03:01:37.876904 28005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260928 03:01:37.877012 28005 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 03:01:37.880062 28005 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 03:01:37.880146 28005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260928 03:01:37.880448 28061 tablet_bootstrap.cc:492] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Bootstrap starting.
I20260928 03:01:37.905436 28061 log.cc:824] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:37.915774 28005 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:44907
I20260928 03:01:37.915807 28168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:44907 every 8 connection(s)
I20260928 03:01:37.916971 28005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:01:37.917325 28061 tablet_bootstrap.cc:492] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:01:37.917975 28061 tablet_bootstrap.cc:492] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Bootstrap complete.
I20260928 03:01:37.919435 28061 ts_tablet_manager.cc:1397] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Time spent bootstrapping tablet: real 0.039s user 0.022s sys 0.015s
I20260928 03:01:37.921518 28171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:37.919617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:57033)
0928 03:01:37.920419 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:37.920663 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":347,"threads_started":1}
I20260928 03:01:37.922166 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 28005
W20260928 03:01:37.922204 28169 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:37675 (0 consecutive failures): Network error: Failed to ping master at 127.12.182.126:37675: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
I20260928 03:01:37.922891 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:37375
--local_ip_for_outbound_sockets=127.12.182.66
--tserver_master_addrs=127.12.182.126:37675
--webserver_port=44137
--webserver_interface=127.12.182.66
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:01:37.923228 28171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:37.922836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:46797)
0928 03:01:37.922995 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:37.923079 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 03:01:37.924203 28171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:37.923794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:51889)
0928 03:01:37.923947 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:37.924045 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 03:01:37.924501 28169 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 03:01:37.926921 28061 raft_consensus.cc:348] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:37.927731 28061 raft_consensus.cc:729] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db6101237d2491bb46fb86536f634d3, State: Initialized, Role: FOLLOWER
I20260928 03:01:37.928272 28061 consensus_queue.cc:260] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:37.929764 28061 ts_tablet_manager.cc:1428] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260928 03:01:37.929771 28171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:37.929393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:57471)
0928 03:01:37.929542 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:37.929642 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 03:01:37.930624 28171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:37.930297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:52833)
0928 03:01:37.930422 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:37.930492 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 03:01:38.230152 28173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:38.230388 28173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:38.230486 28173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:38.237416 28173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:38.237511 28173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:38.237660 28173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:01:38.246968 28173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:37375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=44137
--tserver_master_addrs=127.12.182.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:38.248050 28173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:38.249218 28173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:38.261126 28181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:38.261160 28179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:38.261300 28178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:38.261827 28173 server_base.cc:1034] running on GCE node
I20260928 03:01:38.262367 28173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:38.263170 28173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:38.264351 28173 hybrid_clock.cc:648] HybridClock initialized: now 1790564498264298 us; error 43 us; skew 500 ppm
I20260928 03:01:38.264616 28173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:38 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564498, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:38.267395 28173 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:01:38.268631 28173 webserver.cc:460] Webserver started at http://127.12.182.66:44137/ using document root <none> and password file <none>
I20260928 03:01:38.269223 28173 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:38.269344 28173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:38.274266 28173 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:01:38.276692 28188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:38.277905 28173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 03:01:38.278051 28173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0"
format_stamp: "Formatted at 2026-09-28 03:01:32 on dist-test-slave-k5rg"
server_key: "1e02d12110499b268f32676e45693079"
server_key_iv: "a102abdd031747b469e2ff7c89ab46e6"
server_key_version: "encryptionkey@0"
I20260928 03:01:38.278951 28173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:38.288965 28173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:38.292333 28173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:38.292558 28173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:38.293392 28173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:38.294648 28173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:38.294739 28173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:38.294840 28173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:38.294909 28173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:38.323390 28173 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:37375
I20260928 03:01:38.323457 28300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:37375 every 8 connection(s)
I20260928 03:01:38.324725 28173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:01:38.329032 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:38.327355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:52265)
0928 03:01:38.328043 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:38.328232 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":267,"threads_started":1}
W20260928 03:01:38.329627 28301 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:37675 (0 consecutive failures): Network error: Failed to ping master at 127.12.182.126:37675: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
I20260928 03:01:38.330382 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:38.330078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:45949)
0928 03:01:38.330215 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:38.330285 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 03:01:38.331177 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:38.330884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:48891)
0928 03:01:38.331008 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:38.331077 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 03:01:38.331382 28301 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 03:01:38.333169 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 28173
I20260928 03:01:38.333775 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:42513
--local_ip_for_outbound_sockets=127.12.182.67
--tserver_master_addrs=127.12.182.126:37675
--webserver_port=32865
--webserver_interface=127.12.182.67
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:38.637977 28304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:38.638207 28304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:38.638338 28304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:38.645251 28304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:38.645377 28304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:38.645538 28304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:01:38.654623 28304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:42513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=32865
--tserver_master_addrs=127.12.182.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:38.655715 28304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:38.656875 28304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:38.668752 28309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:38.668804 28312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:38.668783 28310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:38.669415 28304 server_base.cc:1034] running on GCE node
I20260928 03:01:38.670007 28304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:38.670841 28304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:38.672020 28304 hybrid_clock.cc:648] HybridClock initialized: now 1790564498671971 us; error 43 us; skew 500 ppm
I20260928 03:01:38.672282 28304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:38 dist-test-slave-k5rg krb5kdc[26858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564498, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:38.675079 28304 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:01:38.676354 28304 webserver.cc:460] Webserver started at http://127.12.182.67:32865/ using document root <none> and password file <none>
I20260928 03:01:38.676947 28304 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:38.677050 28304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:38.681958 28304 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:01:38.684443 28319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:38.685657 28304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 03:01:38.685804 28304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "7a7148b3104c42138425b9b052484de5"
format_stamp: "Formatted at 2026-09-28 03:01:33 on dist-test-slave-k5rg"
server_key: "5709d05926ca05aee62759d615355b78"
server_key_iv: "7217f16bef8198b9a2e0a3d230826a27"
server_key_version: "encryptionkey@0"
I20260928 03:01:38.686730 28304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:38.717140 28304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:38.720914 28304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:38.721163 28304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:38.722131 28304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:38.723417 28304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:38.723508 28304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:38.723621 28304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:38.723695 28304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:38.752238 28304 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:42513
I20260928 03:01:38.752308 28431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:42513 every 8 connection(s)
I20260928 03:01:38.753549 28304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:01:38.754904 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 28304
I20260928 03:01:38.758095 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:38.756260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:46351)
0928 03:01:38.756986 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:38.757220 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":303,"threads_started":1}
W20260928 03:01:38.758692 28432 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:37675 (0 consecutive failures): Network error: Failed to ping master at 127.12.182.126:37675: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
I20260928 03:01:38.759404 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:38.759110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:41205)
0928 03:01:38.759224 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:38.759287 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 03:01:38.760169 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:38.759866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:54831)
0928 03:01:38.759995 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:38.760066 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 03:01:38.760389 28432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 03:01:38.932080 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:38.931407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:44555)
0928 03:01:38.931810 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:38.931898 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":191,"threads_started":1}
I20260928 03:01:39.332647 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:39.331843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:48489)
0928 03:01:39.332323 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:39.332437 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":224,"threads_started":1}
I20260928 03:01:39.714550 28438 raft_consensus.cc:482] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:01:39.714802 28438 raft_consensus.cc:504] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:39.716265 28438 leader_election.cc:288] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375), 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513)
Sep 28 03:01:39 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564497, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
Sep 28 03:01:39 dist-test-slave-k5rg krb5kdc[26858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564497, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 03:01:39.736038 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:39.716532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:42513 (local address 127.12.182.65:35453)
0928 03:01:39.719010 (+ 2478us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:39.719028 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:39.719140 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:39.721254 (+ 2114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:39.721271 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:39.722068 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:39.722737 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:39.722755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:39.724190 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:39.724194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:39.725762 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:39.725775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:39.726672 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:39.726679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:39.726923 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:39.727588 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:39.727614 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:39.729785 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:39.732883 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:39.732892 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:39.732904 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:39.733288 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:39.733684 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:39.733690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:39.733694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:39.733867 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:39.734549 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:39.734557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:39.734820 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 03:01:39.735036 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 03:01:39.735578 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2368,"mutex_wait_us":367,"spinlock_wait_cycles":24704,"thread_start_us":230,"threads_started":1}
I20260928 03:01:39.736104 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:39.716840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:35453 (local address 127.12.182.67:42513)
0928 03:01:39.719299 (+ 2459us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:39.719309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:39.719337 (+ 28us) server_negotiation.cc:407] Connection header received
0928 03:01:39.720685 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:39.720693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:39.721012 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:39.721269 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:39.722911 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:39.724020 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:39.725886 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:39.726481 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:39.726638 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:39.730045 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:39.730052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:39.730064 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:39.730096 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:39.732752 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:39.733414 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:39.733420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:39.733425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:39.733574 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:39.733972 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:39.733975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:39.733977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:39.734421 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:39.734587 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:39.735284 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 03:01:39.735691 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2310,"thread_start_us":2193,"threads_started":1}
I20260928 03:01:39.736121 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:39.716527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:37375 (local address 127.12.182.65:43573)
0928 03:01:39.717249 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:39.717313 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:39.717928 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:39.721172 (+ 3244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:39.721184 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:39.722067 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:39.722740 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:39.722757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:39.724199 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:39.724203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:39.725765 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:39.725777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:39.726714 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:39.726722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:39.726923 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:39.727644 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:39.727661 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:39.730099 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:39.733273 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:39.733278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:39.733282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:39.733631 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:39.734006 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:39.734012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:39.734016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:39.734156 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:39.734821 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:39.734826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:39.734979 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 03:01:39.735110 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 03:01:39.735728 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":294,"spinlock_wait_cycles":15872,"thread_start_us":2155,"threads_started":1}
I20260928 03:01:39.736992 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:39.716736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:43573 (local address 127.12.182.66:37375)
0928 03:01:39.718746 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:39.718762 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:39.718820 (+ 58us) server_negotiation.cc:407] Connection header received
0928 03:01:39.720404 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:39.720413 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:39.720747 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:39.721027 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:39.722910 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:39.724017 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:39.725889 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:39.726482 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:39.726647 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:39.730333 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:39.730340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:39.730351 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:39.730391 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:39.733146 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:39.733746 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:39.733751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:39.733755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:39.733899 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:39.734259 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:39.734262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:39.734264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:39.734699 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:39.734858 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:39.736333 (+ 1475us) server_negotiation.cc:299] Negotiation successful
0928 03:01:39.736634 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1806,"thread_start_us":1637,"threads_started":1}
W20260928 03:01:39.739550 28055 leader_election.cc:341] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513): Not found: Tablet not found: 3d9f8a68a8f0410492ebeb79a4d56f64
W20260928 03:01:39.739784 28057 leader_election.cc:341] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375): Not found: Tablet not found: 3d9f8a68a8f0410492ebeb79a4d56f64
I20260928 03:01:39.739903 28057 leader_election.cc:302] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1db6101237d2491bb46fb86536f634d3; no voters: 7a7148b3104c42138425b9b052484de5, f99bc8617f7c4d118ebfa7fd224a9ca0
I20260928 03:01:39.740258 28438 raft_consensus.cc:2726] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:01:39.761650 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:39.760839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:34321)
0928 03:01:39.761258 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:39.761396 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":202,"threads_started":1}
I20260928 03:01:39.933290 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:39.932867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:51787)
0928 03:01:39.933030 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:39.933145 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 03:01:40.334069 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:40.333423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:51655)
0928 03:01:40.333841 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:40.333922 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":235,"threads_started":1}
I20260928 03:01:40.763075 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:40.762401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:39207)
0928 03:01:40.762808 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:40.762914 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":210,"threads_started":1}
I20260928 03:01:40.934772 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:40.934061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:37555)
0928 03:01:40.934523 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:40.934602 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":204,"threads_started":1}
I20260928 03:01:41.335469 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:41.334839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:55465)
0928 03:01:41.335233 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:41.335309 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":216,"threads_started":1}
I20260928 03:01:41.382156 28448 raft_consensus.cc:482] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:01:41.382368 28448 raft_consensus.cc:504] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:41.383018 28448 leader_election.cc:288] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375), 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513)
W20260928 03:01:41.384009 28055 leader_election.cc:341] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513): Not found: Tablet not found: 3d9f8a68a8f0410492ebeb79a4d56f64
W20260928 03:01:41.384212 28057 leader_election.cc:341] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375): Not found: Tablet not found: 3d9f8a68a8f0410492ebeb79a4d56f64
I20260928 03:01:41.384316 28057 leader_election.cc:302] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1db6101237d2491bb46fb86536f634d3; no voters: 7a7148b3104c42138425b9b052484de5, f99bc8617f7c4d118ebfa7fd224a9ca0
I20260928 03:01:41.384513 28448 raft_consensus.cc:2726] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:01:41.764437 28449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:41.763806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:35269)
0928 03:01:41.764211 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:41.764285 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":207,"threads_started":1}
I20260928 03:01:41.936192 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:41.935545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:52467)
0928 03:01:41.935974 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:41.936051 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":215,"threads_started":1}
I20260928 03:01:42.336941 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:42.336247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:55677)
0928 03:01:42.336652 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:42.336768 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":189,"threads_started":1}
I20260928 03:01:42.765904 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:42.765204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:38879)
0928 03:01:42.765654 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:42.765732 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":208,"threads_started":1}
I20260928 03:01:42.938045 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:42.937202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:40151)
0928 03:01:42.937701 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:42.937808 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":249,"threads_started":1}
I20260928 03:01:43.293833 28454 raft_consensus.cc:482] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:01:43.294072 28454 raft_consensus.cc:504] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:43.294718 28454 leader_election.cc:288] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375), 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513)
W20260928 03:01:43.295612 28057 leader_election.cc:341] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375): Not found: Tablet not found: 3d9f8a68a8f0410492ebeb79a4d56f64
W20260928 03:01:43.295826 28055 leader_election.cc:341] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513): Not found: Tablet not found: 3d9f8a68a8f0410492ebeb79a4d56f64
I20260928 03:01:43.296002 28055 leader_election.cc:302] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1db6101237d2491bb46fb86536f634d3; no voters: 7a7148b3104c42138425b9b052484de5, f99bc8617f7c4d118ebfa7fd224a9ca0
I20260928 03:01:43.296233 28454 raft_consensus.cc:2726] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:01:43.338554 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:43.337867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:50831)
0928 03:01:43.338295 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:43.338391 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":219,"threads_started":1}
I20260928 03:01:43.767432 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:43.766726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:52793)
0928 03:01:43.767188 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:43.767271 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":235,"threads_started":1}
I20260928 03:01:43.939719 28457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:43.938972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:44363)
0928 03:01:43.939452 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:43.939554 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":244,"threads_started":1}
I20260928 03:01:44.342290 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:44.339313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:55755)
0928 03:01:44.342029 (+ 2716us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:44.342111 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2611,"thread_start_us":2470,"threads_started":1}
I20260928 03:01:44.768877 28459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:44.768174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:53677)
0928 03:01:44.768595 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:44.768673 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":218,"threads_started":1}
I20260928 03:01:44.941344 28460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:44.940536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:52483)
0928 03:01:44.940975 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:44.941078 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":226,"threads_started":1}
I20260928 03:01:45.344026 28461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:45.343168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:45581)
0928 03:01:45.343746 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:45.343845 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":274,"threads_started":1}
I20260928 03:01:45.574908 28462 raft_consensus.cc:482] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:01:45.575137 28462 raft_consensus.cc:504] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:45.575780 28462 leader_election.cc:288] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375), 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513)
W20260928 03:01:45.576651 28057 leader_election.cc:341] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375): Not found: Tablet not found: 3d9f8a68a8f0410492ebeb79a4d56f64
W20260928 03:01:45.576818 28055 leader_election.cc:341] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513): Not found: Tablet not found: 3d9f8a68a8f0410492ebeb79a4d56f64
I20260928 03:01:45.576936 28055 leader_election.cc:302] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1db6101237d2491bb46fb86536f634d3; no voters: 7a7148b3104c42138425b9b052484de5, f99bc8617f7c4d118ebfa7fd224a9ca0
I20260928 03:01:45.577163 28462 raft_consensus.cc:2726] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:01:45.770500 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:45.769759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:53043)
0928 03:01:45.770239 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:45.770318 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":211,"threads_started":1}
I20260928 03:01:45.942955 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:45.942183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:45391)
0928 03:01:45.942633 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:45.942764 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":239,"threads_started":1}
I20260928 03:01:46.345777 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:46.344957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:52779)
0928 03:01:46.345470 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:46.345553 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":244,"threads_started":1}
I20260928 03:01:46.771916 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:46.771290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:41507)
0928 03:01:46.771674 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:46.771750 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":199,"threads_started":1}
I20260928 03:01:46.944389 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:46.943759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:53479)
0928 03:01:46.944144 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:46.944226 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":199,"threads_started":1}
I20260928 03:01:47.347357 28468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:47.346645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:42345)
0928 03:01:47.347107 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:47.347198 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":214,"threads_started":1}
I20260928 03:01:47.773495 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:47.772746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:42965)
0928 03:01:47.773256 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:47.773343 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":241,"threads_started":1}
I20260928 03:01:47.946091 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:47.945330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:60997)
0928 03:01:47.945839 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:47.945928 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":281,"threads_started":1}
I20260928 03:01:48.349083 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:48.348325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:42309)
0928 03:01:48.348799 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:48.348891 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":254,"threads_started":1}
I20260928 03:01:48.775259 28472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:48.774539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:50797)
0928 03:01:48.774988 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:48.775084 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":233,"threads_started":1}
I20260928 03:01:48.947629 28473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:48.946936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:40141)
0928 03:01:48.947355 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:48.947452 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":205,"threads_started":1}
I20260928 03:01:49.350726 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:49.349983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:53983)
0928 03:01:49.350489 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:49.350568 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":263,"threads_started":1}
I20260928 03:01:49.776762 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:49.776090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:53027)
0928 03:01:49.776514 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:49.776601 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":212,"threads_started":1}
I20260928 03:01:49.949069 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:49.948382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:43059)
0928 03:01:49.948805 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:49.948883 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":222,"threads_started":1}
I20260928 03:01:50.352548 28477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:50.351757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:36195)
0928 03:01:50.352286 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:50.352379 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":273,"threads_started":1}
I20260928 03:01:50.423434 28478 raft_consensus.cc:482] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:01:50.423641 28478 raft_consensus.cc:504] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99bc8617f7c4d118ebfa7fd224a9ca0" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37375 } } peers { permanent_uuid: "1db6101237d2491bb46fb86536f634d3" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44907 } } peers { permanent_uuid: "7a7148b3104c42138425b9b052484de5" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 42513 } }
I20260928 03:01:50.424360 28478 leader_election.cc:288] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375), 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513)
W20260928 03:01:50.425276 28057 leader_election.cc:341] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f99bc8617f7c4d118ebfa7fd224a9ca0 (127.12.182.66:37375): Not found: Tablet not found: 3d9f8a68a8f0410492ebeb79a4d56f64
W20260928 03:01:50.425458 28055 leader_election.cc:341] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a7148b3104c42138425b9b052484de5 (127.12.182.67:42513): Not found: Tablet not found: 3d9f8a68a8f0410492ebeb79a4d56f64
I20260928 03:01:50.425576 28055 leader_election.cc:302] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1db6101237d2491bb46fb86536f634d3; no voters: 7a7148b3104c42138425b9b052484de5, f99bc8617f7c4d118ebfa7fd224a9ca0
I20260928 03:01:50.425822 28478 raft_consensus.cc:2726] T 3d9f8a68a8f0410492ebeb79a4d56f64 P 1db6101237d2491bb46fb86536f634d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:01:50.778421 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:50.777714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:52629)
0928 03:01:50.778157 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:50.778253 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":211,"threads_started":1}
I20260928 03:01:50.950620 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:50.949951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:60183)
0928 03:01:50.950353 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:50.950438 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":186,"threads_started":1}
I20260928 03:01:51.354445 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:51.353690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:56947)
0928 03:01:51.354173 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:51.354270 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":230,"threads_started":1}
I20260928 03:01:51.780362 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:51.779442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:46283)
0928 03:01:51.779982 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:51.780079 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":281,"threads_started":1}
I20260928 03:01:51.952291 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:51.951494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:40689)
0928 03:01:51.952004 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:51.952096 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":234,"threads_started":1}
I20260928 03:01:52.356057 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:52.355334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:46541)
0928 03:01:52.355766 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:52.355879 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":230,"threads_started":1}
I20260928 03:01:52.782637 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:52.781709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:41763)
0928 03:01:52.782319 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:52.782414 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":270,"threads_started":1}
I20260928 03:01:52.954116 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:52.953338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.65:36473)
0928 03:01:52.953855 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:52.953944 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":244,"threads_started":1}
I20260928 03:01:53.357779 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:53.357029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.66:42045)
0928 03:01:53.357488 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:53.357587 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":220,"threads_started":1}
I20260928 03:01:53.759907 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 28005
I20260928 03:01:53.774542 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:53.760875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.12.182.65:44907)
0928 03:01:53.761482 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:53.761493 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:53.761523 (+ 30us) server_negotiation.cc:407] Connection header received
0928 03:01:53.761624 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:53.761634 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:53.761902 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:53.762060 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:53.764251 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:53.765269 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:53.766507 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:53.767061 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:53.767158 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:53.769486 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:53.769496 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:53.769510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:53.769541 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:53.772199 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:53.772833 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:53.772839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:53.772841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:53.772975 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:53.773250 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:53.773253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:53.773254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:53.773577 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:53.773845 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:53.774026 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 03:01:53.774212 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":301,"threads_started":1}
I20260928 03:01:53.775679 28164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:53.784472 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:53.783703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:37675 (local address 127.12.182.67:35411)
0928 03:01:53.784214 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:53.784316 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":268,"threads_started":1}
I20260928 03:01:53.863502 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 28005
I20260928 03:01:53.877712 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 28173
I20260928 03:01:53.898265 28491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:53.878455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.12.182.66:37375)
0928 03:01:53.878894 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:53.878899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:53.878915 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:01:53.878968 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:53.878972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:53.879031 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:53.879118 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:53.880409 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:53.881237 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:53.887754 (+ 6517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:53.888509 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:53.888688 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:53.891882 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:53.891889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:53.891893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:53.891946 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:53.894968 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:53.895811 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:53.895816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:53.895818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:53.895989 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:53.896456 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:53.896461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:53.896463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:53.896788 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:53.897059 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:53.897443 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 03:01:53.897917 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":228,"threads_started":1}
I20260928 03:01:53.899076 28296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:53.985244 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 28173
I20260928 03:01:53.999696 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 28304
I20260928 03:01:54.011966 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:54.000447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.12.182.67:42513)
0928 03:01:54.000979 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:54.000984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:54.001001 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:01:54.001055 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:54.001058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:54.001117 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:54.001207 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:54.002611 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:54.003381 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:54.004534 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:54.004935 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:54.005026 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:54.007396 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:54.007401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:54.007405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:54.007432 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:54.009817 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:54.010393 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:54.010396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:54.010399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:54.010546 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:54.010828 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:54.010830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:54.010832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:54.011062 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:54.011259 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:54.011452 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 03:01:54.011607 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":262,"threads_started":1}
I20260928 03:01:54.012630 28427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:54.095028 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 28304
2026-09-28T03:01:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23893 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:01:54 dist-test-slave-k5rg krb5kdc[28497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:01:54 dist-test-slave-k5rg krb5kdc[28497](info): set up 2 sockets
Sep 28 03:01:54 dist-test-slave-k5rg krb5kdc[28497](info): commencing operation
krb5kdc: starting...
W20260928 03:01:55.105883 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:01:55 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:01:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:01:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:01:55.317274 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:42827
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:42827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:55.621361 28510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:55.621681 28510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:55.621830 28510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:55.628939 28510 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 03:01:55.629034 28510 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 03:01:55.629092 28510 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 03:01:55.629163 28510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:01:55.629230 28510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:55.629328 28510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:55.629385 28510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:01:55.629462 28510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:01:55.639142 28510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:42827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:42827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:55.640353 28510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:55.641667 28510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:55.653018 28515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:55.652972 28516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:55.653198 28510 server_base.cc:1034] running on GCE node
W20260928 03:01:55.652925 28518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:55.654207 28510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:55.655474 28510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:55.656682 28510 hybrid_clock.cc:648] HybridClock initialized: now 1790564515656623 us; error 32 us; skew 500 ppm
I20260928 03:01:55.656986 28510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:55 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564515, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:55.659884 28510 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:01:55.661167 28510 webserver.cc:460] Webserver started at http://127.12.182.126:34939/ using document root <none> and password file <none>
I20260928 03:01:55.661839 28510 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:55.661947 28510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:55.662220 28510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:55.665023 28510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "2f59068f840e4274b8661ca5781d9626"
format_stamp: "Formatted at 2026-09-28 03:01:55 on dist-test-slave-k5rg"
server_key: "a9b56fafb660ead950af051377b83a25"
server_key_iv: "82ba53398c19d41faaf8b7d0131ff574"
server_key_version: "encryptionkey@0"
I20260928 03:01:55.665774 28510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "2f59068f840e4274b8661ca5781d9626"
format_stamp: "Formatted at 2026-09-28 03:01:55 on dist-test-slave-k5rg"
server_key: "a9b56fafb660ead950af051377b83a25"
server_key_iv: "82ba53398c19d41faaf8b7d0131ff574"
server_key_version: "encryptionkey@0"
I20260928 03:01:55.670434 28510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:01:55.673512 28525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:55.674749 28510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 03:01:55.674888 28510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "2f59068f840e4274b8661ca5781d9626"
format_stamp: "Formatted at 2026-09-28 03:01:55 on dist-test-slave-k5rg"
server_key: "a9b56fafb660ead950af051377b83a25"
server_key_iv: "82ba53398c19d41faaf8b7d0131ff574"
server_key_version: "encryptionkey@0"
I20260928 03:01:55.675073 28510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:55.700560 28510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:55.704566 28510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:55.704838 28510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:55.720956 28510 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:42827
I20260928 03:01:55.720980 28576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:42827 every 8 connection(s)
I20260928 03:01:55.722265 28510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:01:55.725375 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 28510
I20260928 03:01:55.725560 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:01:55.725729 28577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:55.726258 13017 external_mini_cluster.cc:1428] Setting key 839f45859c4ac0f37a852f395d92100f
I20260928 03:01:55.737111 28577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Bootstrap starting.
I20260928 03:01:55.739974 28577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:55.740888 28577 log.cc:824] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:55.743355 28577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: No bootstrap required, opened a new log
Sep 28 03:01:55 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:55.753190 28577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:01:55.753657 28577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:55.753665 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:55.728159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.12.182.126:42827)
0928 03:01:55.728930 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:55.728944 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:55.728990 (+ 46us) server_negotiation.cc:407] Connection header received
0928 03:01:55.731501 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:55.731522 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:55.731894 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:55.732235 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:55.733661 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:55.734935 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:55.736200 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:55.736830 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:55.736996 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:55.745396 (+ 8400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:55.745407 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:55.745424 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:55.745466 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:55.749300 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:55.750022 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:55.750030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:55.750038 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:55.750248 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:55.750590 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:55.750595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:55.750599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:55.751297 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:55.751597 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:55.751910 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 03:01:55.752331 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":298,"threads_started":1}
I20260928 03:01:55.753786 28577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f59068f840e4274b8661ca5781d9626, State: Initialized, Role: FOLLOWER
I20260928 03:01:55.754369 28577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:01:55.754558 28577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:01:55.754648 28577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:01:55.754757 28577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:55.757025 28577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:01:55.757467 28577 leader_election.cc:302] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f59068f840e4274b8661ca5781d9626; no voters:
I20260928 03:01:55.757961 28577 leader_election.cc:288] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:01:55.758080 28582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:01:55.758678 28582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 1 LEADER]: Becoming Leader. State: Replica: 2f59068f840e4274b8661ca5781d9626, State: Running, Role: LEADER
I20260928 03:01:55.759186 28582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:01:55.759819 28577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:01:55.762252 28584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f59068f840e4274b8661ca5781d9626. Latest consensus state: current_term: 1 leader_uuid: "2f59068f840e4274b8661ca5781d9626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } } }
I20260928 03:01:55.762230 28583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f59068f840e4274b8661ca5781d9626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } } }
I20260928 03:01:55.762650 28584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:55.762646 28583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: This master's current role is: LEADER
I20260928 03:01:55.764187 28591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:01:55.768944 28591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:01:55.777629 28591 catalog_manager.cc:1347] Generated new cluster ID: ecc95aeffb2a4e6e87820c0d3011c284
I20260928 03:01:55.777760 28591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:01:55.822826 28591 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:01:55.824095 28591 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:01:55.836570 28591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Generated new TSK 0
I20260928 03:01:55.837558 28591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 03:01:55.890931 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:42827
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:56.201651 28603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:56.201900 28603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:56.202037 28603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:56.209033 28603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:56.209126 28603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:56.209272 28603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:01:56.218578 28603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:56.219772 28603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:56.221028 28603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:56.233096 28608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:56.233028 28611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:56.233067 28609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:56.234086 28603 server_base.cc:1034] running on GCE node
I20260928 03:01:56.234663 28603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:56.235555 28603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:56.236750 28603 hybrid_clock.cc:648] HybridClock initialized: now 1790564516236710 us; error 32 us; skew 500 ppm
I20260928 03:01:56.237035 28603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:56 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:56.239928 28603 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:01:56.241226 28603 webserver.cc:460] Webserver started at http://127.12.182.65:43583/ using document root <none> and password file <none>
I20260928 03:01:56.241906 28603 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:56.242023 28603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:56.242295 28603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:56.244973 28603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "1c8e82f9763c4e8f966b96eaf588fb2f"
format_stamp: "Formatted at 2026-09-28 03:01:56 on dist-test-slave-k5rg"
server_key: "7dcf0b66051160627017da086ba8a3e1"
server_key_iv: "9e907f77bdc309eb2f31a9ece32088c5"
server_key_version: "encryptionkey@0"
I20260928 03:01:56.245702 28603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "1c8e82f9763c4e8f966b96eaf588fb2f"
format_stamp: "Formatted at 2026-09-28 03:01:56 on dist-test-slave-k5rg"
server_key: "7dcf0b66051160627017da086ba8a3e1"
server_key_iv: "9e907f77bdc309eb2f31a9ece32088c5"
server_key_version: "encryptionkey@0"
I20260928 03:01:56.250274 28603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:01:56.253463 28618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:56.254688 28603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:01:56.254832 28603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "1c8e82f9763c4e8f966b96eaf588fb2f"
format_stamp: "Formatted at 2026-09-28 03:01:56 on dist-test-slave-k5rg"
server_key: "7dcf0b66051160627017da086ba8a3e1"
server_key_iv: "9e907f77bdc309eb2f31a9ece32088c5"
server_key_version: "encryptionkey@0"
I20260928 03:01:56.254998 28603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:56.268831 28603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:56.272347 28603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:56.272593 28603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:56.273433 28603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:56.274623 28603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:56.274698 28603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:56.274804 28603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:56.274859 28603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:56.304180 28603 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:39447
I20260928 03:01:56.304196 28730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:39447 every 8 connection(s)
I20260928 03:01:56.305382 28603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:01:56.310590 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 28603
I20260928 03:01:56.310775 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 03:01:56.311396 13017 external_mini_cluster.cc:1428] Setting key 57e5214c2f3b4a485a3df022418289cb
Sep 28 03:01:56 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:56.324303 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:56.307989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:38805 (local address 127.12.182.126:42827)
0928 03:01:56.308447 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:56.308453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:56.309353 (+ 900us) server_negotiation.cc:407] Connection header received
0928 03:01:56.311196 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:56.311202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:56.311279 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:56.311372 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:56.312933 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:56.313810 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:56.315310 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:56.315698 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:56.315806 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:56.318997 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:56.319001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:56.319004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:56.319037 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:56.321468 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:56.322135 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:56.322139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:56.322141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:56.322256 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:56.322663 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:56.322666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:56.322668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:56.322938 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:56.323066 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:56.323723 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 03:01:56.323950 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":205,"threads_started":1}
I20260928 03:01:56.325142 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:56.308517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:38805)
0928 03:01:56.309192 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:56.309234 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:56.310944 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:56.311619 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:56.311628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:56.312223 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:56.312759 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:56.312776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:56.313955 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:56.313959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:56.315168 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:56.315180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:56.315904 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:56.315911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:56.316145 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:56.316752 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:56.316773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:56.318789 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:56.321595 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:56.321622 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:56.321637 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:56.322022 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:56.322367 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:56.322372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:56.322376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:56.322550 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:56.323059 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:56.323066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:56.323371 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 03:01:56.323586 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 03:01:56.324080 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":250,"threads_started":1}
I20260928 03:01:56.326694 28731 heartbeater.cc:347] Connected to a master server at 127.12.182.126:42827
I20260928 03:01:56.326997 28731 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:56.327831 28731 heartbeater.cc:511] Master 127.12.182.126:42827 requested a full tablet report, sending...
I20260928 03:01:56.330277 28542 ts_manager.cc:194] Registered new tserver with Master: 1c8e82f9763c4e8f966b96eaf588fb2f (127.12.182.65:39447)
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
I20260928 03:01:56.332170 28542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:38805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 03:01:56.360260 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:42827
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:56.660776 28737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:56.661036 28737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:56.661166 28737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:56.668768 28737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:56.668893 28737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:56.669049 28737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:01:56.680799 28737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:56.682114 28737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:56.683382 28737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:56.695411 28742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:56.695364 28745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:56.695806 28743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:56.696133 28737 server_base.cc:1034] running on GCE node
I20260928 03:01:56.696698 28737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:56.697569 28737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:56.698784 28737 hybrid_clock.cc:648] HybridClock initialized: now 1790564516698740 us; error 36 us; skew 500 ppm
I20260928 03:01:56.699126 28737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:56 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:56.702081 28737 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:01:56.703491 28737 webserver.cc:460] Webserver started at http://127.12.182.66:42427/ using document root <none> and password file <none>
I20260928 03:01:56.704128 28737 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:56.704253 28737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:56.704524 28737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:56.707329 28737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "a2618ef5787e4758bbbb4adacc63686c"
format_stamp: "Formatted at 2026-09-28 03:01:56 on dist-test-slave-k5rg"
server_key: "be6c5f6dbf163832b2dc4e04fc85cd15"
server_key_iv: "4e1bedd631ef2637c335f2fb5168824e"
server_key_version: "encryptionkey@0"
I20260928 03:01:56.708037 28737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "a2618ef5787e4758bbbb4adacc63686c"
format_stamp: "Formatted at 2026-09-28 03:01:56 on dist-test-slave-k5rg"
server_key: "be6c5f6dbf163832b2dc4e04fc85cd15"
server_key_iv: "4e1bedd631ef2637c335f2fb5168824e"
server_key_version: "encryptionkey@0"
I20260928 03:01:56.713004 28737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:01:56.716311 28752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:56.717556 28737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 03:01:56.717721 28737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "a2618ef5787e4758bbbb4adacc63686c"
format_stamp: "Formatted at 2026-09-28 03:01:56 on dist-test-slave-k5rg"
server_key: "be6c5f6dbf163832b2dc4e04fc85cd15"
server_key_iv: "4e1bedd631ef2637c335f2fb5168824e"
server_key_version: "encryptionkey@0"
I20260928 03:01:56.717904 28737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:56.738327 28737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:56.742183 28737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:56.742429 28737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:56.743337 28737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:56.744563 28737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:56.744640 28737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:56.744757 28737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:56.744819 28737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:56.774066 28737 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:37549
I20260928 03:01:56.774127 28864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:37549 every 8 connection(s)
I20260928 03:01:56.775262 28737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:01:56.779699 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 28737
I20260928 03:01:56.779892 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 03:01:56.780450 13017 external_mini_cluster.cc:1428] Setting key 94467547953c121898f6642ed6afe73f
Sep 28 03:01:56 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:56.796768 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:56.777564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:40731 (local address 127.12.182.126:42827)
0928 03:01:56.777789 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:56.777794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:56.778913 (+ 1119us) server_negotiation.cc:407] Connection header received
0928 03:01:56.781187 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:56.781193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:56.781271 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:56.781364 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:56.783041 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:56.783753 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:56.785814 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:56.786192 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:56.786283 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:56.790633 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:56.790639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:56.790643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:56.790687 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:56.793971 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:56.794654 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:56.794659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:56.794663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:56.794814 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:56.795179 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:56.795183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:56.795185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:56.795490 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:56.795624 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:56.796245 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 03:01:56.796465 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 03:01:56.797562 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:56.778105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:40731)
0928 03:01:56.778769 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:56.778804 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:56.780916 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:56.781595 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:56.781630 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:56.782157 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:56.782868 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:56.782895 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:56.783934 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:56.783941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:56.785688 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:56.785703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:56.786394 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:56.786400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:56.786662 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:56.787380 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:56.787413 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:56.790375 (+ 2962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:56.794103 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:56.794110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:56.794121 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:56.794534 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:56.794900 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:56.794903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:56.794906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:56.795073 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:56.795593 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:56.795598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:56.795906 (+ 308us) client_negotiation.cc:769] Sending connection context
0928 03:01:56.796121 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 03:01:56.796505 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":254,"threads_started":1}
I20260928 03:01:56.798898 28865 heartbeater.cc:347] Connected to a master server at 127.12.182.126:42827
I20260928 03:01:56.799188 28865 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:56.799868 28865 heartbeater.cc:511] Master 127.12.182.126:42827 requested a full tablet report, sending...
I20260928 03:01:56.801265 28542 ts_manager.cc:194] Registered new tserver with Master: a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549)
I20260928 03:01:56.802109 28542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:40731
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 03:01:56.830749 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:42827
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:01:57.139055 28870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:01:57.139338 28870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:01:57.139457 28870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:01:57.146461 28870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:01:57.146569 28870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:01:57.146723 28870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:01:57.156036 28870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:01:57.157260 28870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:01:57.158561 28870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:01:57.171144 28878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:57.171116 28876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:01:57.171121 28875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:01:57.172009 28870 server_base.cc:1034] running on GCE node
I20260928 03:01:57.172613 28870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:01:57.173516 28870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:01:57.174753 28870 hybrid_clock.cc:648] HybridClock initialized: now 1790564517174677 us; error 44 us; skew 500 ppm
I20260928 03:01:57.175051 28870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:01:57 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:01:57.178128 28870 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:01:57.179628 28870 webserver.cc:460] Webserver started at http://127.12.182.67:44485/ using document root <none> and password file <none>
I20260928 03:01:57.180315 28870 fs_manager.cc:359] Metadata directory not provided
I20260928 03:01:57.180429 28870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:01:57.180699 28870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:01:57.183540 28870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "caa2934228e54d89b97ea95c8d5ec0ad"
format_stamp: "Formatted at 2026-09-28 03:01:57 on dist-test-slave-k5rg"
server_key: "11a6b09f038d00aa1c037663610aed88"
server_key_iv: "daf335752013ec22efae2ba3e054094a"
server_key_version: "encryptionkey@0"
I20260928 03:01:57.184284 28870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "caa2934228e54d89b97ea95c8d5ec0ad"
format_stamp: "Formatted at 2026-09-28 03:01:57 on dist-test-slave-k5rg"
server_key: "11a6b09f038d00aa1c037663610aed88"
server_key_iv: "daf335752013ec22efae2ba3e054094a"
server_key_version: "encryptionkey@0"
I20260928 03:01:57.189260 28870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:01:57.192673 28885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:57.194002 28870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:01:57.194145 28870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "caa2934228e54d89b97ea95c8d5ec0ad"
format_stamp: "Formatted at 2026-09-28 03:01:57 on dist-test-slave-k5rg"
server_key: "11a6b09f038d00aa1c037663610aed88"
server_key_iv: "daf335752013ec22efae2ba3e054094a"
server_key_version: "encryptionkey@0"
I20260928 03:01:57.194311 28870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:01:57.220883 28870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:01:57.225113 28870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:01:57.225386 28870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:01:57.226365 28870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:01:57.227584 28870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:01:57.227654 28870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:57.227769 28870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:01:57.227816 28870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:01:57.254577 28870 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:38355
I20260928 03:01:57.254612 28997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:38355 every 8 connection(s)
I20260928 03:01:57.255741 28870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:01:57.260143 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 28870
I20260928 03:01:57.260330 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 03:01:57.260762 13017 external_mini_cluster.cc:1428] Setting key 3b8c9ab529a72a8036295c494b20c7a2
Sep 28 03:01:57 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:01:57.273870 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.258105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:43943 (local address 127.12.182.126:42827)
0928 03:01:57.258307 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.258313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.259576 (+ 1263us) server_negotiation.cc:407] Connection header received
0928 03:01:57.261426 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.261432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.261517 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.261595 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:57.263137 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.264150 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.265726 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.266134 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.266232 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.269167 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:57.269172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:57.269175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:57.269206 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:57.271539 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:57.272144 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:57.272147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:57.272150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:57.272251 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:57.272562 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:57.272565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:57.272567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:57.272808 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:57.272919 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.273419 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.273585 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:01:57.275025 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.258654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:43943)
0928 03:01:57.259406 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:57.259455 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:57.261174 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:57.261854 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:57.261865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:57.262382 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:01:57.262992 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.263010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.264278 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.264282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.265536 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.265552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.266302 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.266307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.266501 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.267029 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:01:57.267050 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:01:57.268994 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:01:57.271657 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:57.271663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:57.271675 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:57.272063 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:57.272325 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:01:57.272328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:01:57.272330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:01:57.272490 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:01:57.272894 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:01:57.272899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:01:57.273161 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 03:01:57.273367 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 03:01:57.273808 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":273,"threads_started":1}
I20260928 03:01:57.276230 28998 heartbeater.cc:347] Connected to a master server at 127.12.182.126:42827
I20260928 03:01:57.276512 28998 heartbeater.cc:464] Registering TS with master...
I20260928 03:01:57.277194 28998 heartbeater.cc:511] Master 127.12.182.126:42827 requested a full tablet report, sending...
I20260928 03:01:57.278503 28542 ts_manager.cc:194] Registered new tserver with Master: caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355)
I20260928 03:01:57.279282 28542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:43943
I20260928 03:01:57.287715 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:01:57.303212 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.291748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.12.182.126:42827)
0928 03:01:57.291933 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.291939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.292334 (+ 395us) server_negotiation.cc:407] Connection header received
0928 03:01:57.292461 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.292464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.292522 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.292592 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:57.293935 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.294616 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.295867 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.296242 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.296324 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.298514 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:57.298519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:57.298524 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:57.298562 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:57.300748 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:57.301358 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:57.301361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:57.301363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:57.301465 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:57.301925 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:57.301930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:57.301932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:57.302241 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:57.302438 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.302772 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.302924 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 03:01:57.307168 28542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:01:57.313630 28542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:01:57.335371 28731 heartbeater.cc:503] Master 127.12.182.126:42827 was elected leader, sending a full tablet report...
I20260928 03:01:57.348429 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.326393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:38355 (local address 127.0.0.1:36482)
0928 03:01:57.339686 (+ 13293us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:57.339705 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:57.339826 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:57.340587 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:57.340594 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:57.340623 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:57.340999 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.341011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.342952 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.342957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.345365 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.345379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.347413 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.347430 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.347618 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.347668 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 03:01:57.347799 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 03:01:57.347984 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13158,"mutex_wait_us":61,"thread_start_us":13042,"threads_started":1}
I20260928 03:01:57.348670 29013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.325964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.12.182.65:39447)
0928 03:01:57.341694 (+ 15730us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.341702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.341726 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:01:57.341811 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.341818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.342018 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.342881 (+ 863us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:57.343103 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.344502 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.346450 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.347672 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.347787 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.347988 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.348212 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.348380 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15555,"thread_start_us":15402,"threads_started":1}
I20260928 03:01:57.348779 29012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.325350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.0.0.1:60494)
0928 03:01:57.338109 (+ 12759us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:57.338134 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:57.338308 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:57.342233 (+ 3925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:57.342238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:57.342258 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:57.342683 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.342696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.344634 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.344639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.346291 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.346303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.347880 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.347887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.348049 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.348076 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 03:01:57.348155 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 03:01:57.348287 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12137,"mutex_wait_us":2,"thread_start_us":11996,"threads_started":1}
I20260928 03:01:57.348850 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.325161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:37549 (local address 127.0.0.1:52254)
0928 03:01:57.326056 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:57.326090 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:57.326242 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:57.341834 (+ 15592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:57.341840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:57.341866 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:57.342261 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.342273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.344270 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.344276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.346381 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.346396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.348097 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.348106 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.348272 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.348302 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 03:01:57.348371 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 03:01:57.348488 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":23,"thread_start_us":535,"threads_started":1}
I20260928 03:01:57.349274 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.326789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.12.182.67:38355)
0928 03:01:57.339081 (+ 12292us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.339092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.339967 (+ 875us) server_negotiation.cc:407] Connection header received
0928 03:01:57.340085 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.340093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.340371 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.340573 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:57.341195 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.342793 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.345534 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.346946 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.347062 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.347281 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.348808 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.348965 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12084,"thread_start_us":12311,"threads_started":1}
I20260928 03:01:57.350039 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.325345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.12.182.66:37549)
0928 03:01:57.341241 (+ 15896us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.341250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.341276 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:01:57.341363 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.341370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.341598 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.342459 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:57.342666 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.344129 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.346555 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.347874 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.348969 (+ 1095us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.349412 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.349551 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.349733 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15731,"thread_start_us":15621,"threads_started":1}
I20260928 03:01:57.352595 28666 tablet_service.cc:1467] Processing CreateTablet for tablet aa7ddc9bfa7342fe92896c83696c11ae (DEFAULT_TABLE table=test-table [id=0cb96dff028a417386a044bb0937a6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:57.352732 28933 tablet_service.cc:1467] Processing CreateTablet for tablet aa7ddc9bfa7342fe92896c83696c11ae (DEFAULT_TABLE table=test-table [id=0cb96dff028a417386a044bb0937a6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:57.353045 28800 tablet_service.cc:1467] Processing CreateTablet for tablet aa7ddc9bfa7342fe92896c83696c11ae (DEFAULT_TABLE table=test-table [id=0cb96dff028a417386a044bb0937a6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:01:57.353855 28666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7ddc9bfa7342fe92896c83696c11ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:57.354094 28933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7ddc9bfa7342fe92896c83696c11ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:57.354151 28800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7ddc9bfa7342fe92896c83696c11ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:01:57.367508 29017 tablet_bootstrap.cc:492] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: Bootstrap starting.
I20260928 03:01:57.370271 29018 tablet_bootstrap.cc:492] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: Bootstrap starting.
I20260928 03:01:57.372069 29017 tablet_bootstrap.cc:654] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:57.372838 29018 tablet_bootstrap.cc:654] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:57.373345 29017 log.cc:824] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:57.373582 29016 tablet_bootstrap.cc:492] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: Bootstrap starting.
I20260928 03:01:57.374648 29018 log.cc:824] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:57.376606 29017 tablet_bootstrap.cc:492] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: No bootstrap required, opened a new log
I20260928 03:01:57.376750 29016 tablet_bootstrap.cc:654] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: Neither blocks nor log segments found. Creating new log.
I20260928 03:01:57.376843 29017 ts_tablet_manager.cc:1397] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 03:01:57.377498 29018 tablet_bootstrap.cc:492] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: No bootstrap required, opened a new log
I20260928 03:01:57.377823 29018 ts_tablet_manager.cc:1397] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 03:01:57.377952 29016 log.cc:824] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: Log is configured to *not* fsync() on all Append() calls
I20260928 03:01:57.384394 29016 tablet_bootstrap.cc:492] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: No bootstrap required, opened a new log
I20260928 03:01:57.384642 29016 ts_tablet_manager.cc:1397] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260928 03:01:57.384619 29017 raft_consensus.cc:348] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } } peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } }
I20260928 03:01:57.384969 29017 raft_consensus.cc:374] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:57.385067 29017 raft_consensus.cc:729] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2618ef5787e4758bbbb4adacc63686c, State: Initialized, Role: FOLLOWER
I20260928 03:01:57.385643 29017 consensus_queue.cc:260] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } } peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } }
I20260928 03:01:57.386641 28865 heartbeater.cc:503] Master 127.12.182.126:42827 was elected leader, sending a full tablet report...
I20260928 03:01:57.387086 29017 ts_tablet_manager.cc:1428] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260928 03:01:57.389142 29018 raft_consensus.cc:348] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } } peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } }
I20260928 03:01:57.389623 29018 raft_consensus.cc:374] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:57.389783 29018 raft_consensus.cc:729] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa2934228e54d89b97ea95c8d5ec0ad, State: Initialized, Role: FOLLOWER
I20260928 03:01:57.390379 29018 consensus_queue.cc:260] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } } peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } }
I20260928 03:01:57.395270 29016 raft_consensus.cc:348] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } } peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } }
I20260928 03:01:57.395682 29016 raft_consensus.cc:374] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:01:57.395788 29016 raft_consensus.cc:729] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8e82f9763c4e8f966b96eaf588fb2f, State: Initialized, Role: FOLLOWER
I20260928 03:01:57.395869 28998 heartbeater.cc:503] Master 127.12.182.126:42827 was elected leader, sending a full tablet report...
I20260928 03:01:57.396368 29016 consensus_queue.cc:260] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } } peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } }
I20260928 03:01:57.397838 29018 ts_tablet_manager.cc:1428] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: Time spent starting tablet: real 0.020s user 0.017s sys 0.000s
I20260928 03:01:57.397920 29016 ts_tablet_manager.cc:1428] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: Time spent starting tablet: real 0.013s user 0.012s sys 0.003s
I20260928 03:01:57.489831 29024 raft_consensus.cc:482] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:01:57.490129 29024 raft_consensus.cc:504] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } } peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } }
I20260928 03:01:57.491860 29024 leader_election.cc:288] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549), caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355)
I20260928 03:01:57.499492 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.492221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:48961 (local address 127.12.182.66:37549)
0928 03:01:57.492429 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.492434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.492960 (+ 526us) server_negotiation.cc:407] Connection header received
0928 03:01:57.493176 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.493181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.493260 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.493349 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:57.494103 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.495041 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.497449 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.498638 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.498787 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.498857 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.499139 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.499233 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 03:01:57.499620 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.493244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:39417 (local address 127.12.182.67:38355)
0928 03:01:57.493430 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.493435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.493726 (+ 291us) server_negotiation.cc:407] Connection header received
0928 03:01:57.493883 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.493886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.493949 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.494014 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:01:57.494498 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.495409 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.497657 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.498822 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.498953 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.499000 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.499291 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.499408 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:01:57.499653 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.493036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:38355 (local address 127.12.182.65:39417)
0928 03:01:57.493639 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:57.493656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:57.493778 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:57.494054 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:57.494058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:57.494084 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:57.494352 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.494368 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.495531 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.495536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.497481 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.497490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.499001 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.499007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.499157 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.499186 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 03:01:57.499256 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 03:01:57.499354 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":255,"threads_started":1}
I20260928 03:01:57.499681 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.492209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:37549 (local address 127.12.182.65:48961)
0928 03:01:57.492873 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 03:01:57.492892 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:01:57.493025 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:01:57.493517 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:01:57.493522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:01:57.493550 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:01:57.493966 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.493977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.495164 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.495168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.497289 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:01:57.497304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.498807 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.498812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:01:57.498974 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.499013 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 03:01:57.499111 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 03:01:57.499247 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":55,"thread_start_us":361,"threads_started":1}
I20260928 03:01:57.500792 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7ddc9bfa7342fe92896c83696c11ae" candidate_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" is_pre_election: true
I20260928 03:01:57.500936 28820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7ddc9bfa7342fe92896c83696c11ae" candidate_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2618ef5787e4758bbbb4adacc63686c" is_pre_election: true
I20260928 03:01:57.501335 28953 raft_consensus.cc:2454] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c8e82f9763c4e8f966b96eaf588fb2f in term 0.
I20260928 03:01:57.501377 28820 raft_consensus.cc:2454] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c8e82f9763c4e8f966b96eaf588fb2f in term 0.
I20260928 03:01:57.501976 28621 leader_election.cc:302] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8e82f9763c4e8f966b96eaf588fb2f, caa2934228e54d89b97ea95c8d5ec0ad; no voters:
I20260928 03:01:57.502302 29024 raft_consensus.cc:2781] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:01:57.502442 29024 raft_consensus.cc:482] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:01:57.502544 29024 raft_consensus.cc:3037] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:57.504588 29024 raft_consensus.cc:504] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } } peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } }
I20260928 03:01:57.505120 29024 leader_election.cc:288] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 1 election: Requested vote from peers a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549), caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355)
I20260928 03:01:57.505478 28820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7ddc9bfa7342fe92896c83696c11ae" candidate_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2618ef5787e4758bbbb4adacc63686c"
I20260928 03:01:57.505595 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7ddc9bfa7342fe92896c83696c11ae" candidate_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa2934228e54d89b97ea95c8d5ec0ad"
I20260928 03:01:57.505666 28820 raft_consensus.cc:3037] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:01:57.505801 28953 raft_consensus.cc:3037] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad [term 0 FOLLOWER]: Advancing to term 1
W20260928 03:01:57.507280 28999 tablet.cc:2369] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:01:57.507946 28820 raft_consensus.cc:2454] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c8e82f9763c4e8f966b96eaf588fb2f in term 1.
I20260928 03:01:57.508272 28621 leader_election.cc:302] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8e82f9763c4e8f966b96eaf588fb2f, a2618ef5787e4758bbbb4adacc63686c; no voters:
I20260928 03:01:57.508344 28953 raft_consensus.cc:2454] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c8e82f9763c4e8f966b96eaf588fb2f in term 1.
I20260928 03:01:57.508502 29024 raft_consensus.cc:2781] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:01:57.508957 29024 raft_consensus.cc:686] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 LEADER]: Becoming Leader. State: Replica: 1c8e82f9763c4e8f966b96eaf588fb2f, State: Running, Role: LEADER
I20260928 03:01:57.509397 29024 consensus_queue.cc:237] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } } peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } }
I20260928 03:01:57.513401 28541 catalog_manager.cc:5495] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c8e82f9763c4e8f966b96eaf588fb2f (127.12.182.65). New cstate: current_term: 1 leader_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } health_report { overall_health: UNKNOWN } } }
W20260928 03:01:57.527168 28866 tablet.cc:2369] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:01:57.541267 29013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.535701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.12.182.65:39447)
0928 03:01:57.535917 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.535921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.535938 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:01:57.536082 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.536085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.536163 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.536289 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:01:57.536593 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.537382 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.539124 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.539539 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.539633 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.540065 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:01:57.540579 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:01:57.540721 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.540909 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.540995 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 03:01:57.549597 28953 raft_consensus.cc:1261] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad [term 1 FOLLOWER]: Refusing update from remote peer 1c8e82f9763c4e8f966b96eaf588fb2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:01:57.549597 28820 raft_consensus.cc:1261] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c [term 1 FOLLOWER]: Refusing update from remote peer 1c8e82f9763c4e8f966b96eaf588fb2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:01:57.550578 29024 consensus_queue.cc:1035] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:01:57.550944 29027 consensus_queue.cc:1035] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 03:01:57.557845 28732 tablet.cc:2369] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:01:57.561326 29030 mvcc.cc:205] Tried to move back new op lower bound from 7334152263872294912 to 7334152263721848832. Current Snapshot: MvccSnapshot[applied={T|T < 7334152263872294912}]
I20260928 03:01:57.564380 29031 mvcc.cc:205] Tried to move back new op lower bound from 7334152263872294912 to 7334152263721848832. Current Snapshot: MvccSnapshot[applied={T|T < 7334152263872294912}]
I20260928 03:01:57.569016 29032 mvcc.cc:205] Tried to move back new op lower bound from 7334152263872294912 to 7334152263721848832. Current Snapshot: MvccSnapshot[applied={T|T < 7334152263872294912}]
I20260928 03:01:57.574457 28541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:01:57.574683 28541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:01:57.578173 28541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Sending DeleteTablet for 3 replicas of tablet aa7ddc9bfa7342fe92896c83696c11ae
I20260928 03:01:57.579293 28666 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7ddc9bfa7342fe92896c83696c11ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:57 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:60494
I20260928 03:01:57.579304 28800 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7ddc9bfa7342fe92896c83696c11ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:57 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:52254
I20260928 03:01:57.579457 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 28603
I20260928 03:01:57.580147 28933 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7ddc9bfa7342fe92896c83696c11ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:01:57 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:36482
I20260928 03:01:57.590801 29039 tablet_replica.cc:330] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: stopping tablet replica
I20260928 03:01:57.593830 29039 raft_consensus.cc:2229] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:01:57.594823 29040 tablet_replica.cc:330] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: stopping tablet replica
I20260928 03:01:57.595218 29039 raft_consensus.cc:2258] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:57.595266 29040 raft_consensus.cc:2229] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:01:57.595799 29040 raft_consensus.cc:2258] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:57.596029 29042 tablet_replica.cc:330] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: stopping tablet replica
I20260928 03:01:57.597054 29042 raft_consensus.cc:2229] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:01:57.597528 29042 raft_consensus.cc:2258] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:01:57.597986 29040 ts_tablet_manager.cc:1905] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:01:57.599483 29042 ts_tablet_manager.cc:1905] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 03:01:57 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 03:01:57.604151 29039 ts_tablet_manager.cc:1905] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:01:57.605047 29040 ts_tablet_manager.cc:1918] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:01:57.605242 29040 log.cc:1196] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/aa7ddc9bfa7342fe92896c83696c11ae
I20260928 03:01:57.605829 29040 ts_tablet_manager.cc:1939] T aa7ddc9bfa7342fe92896c83696c11ae P a2618ef5787e4758bbbb4adacc63686c: Deleting consensus metadata
I20260928 03:01:57.606045 29042 ts_tablet_manager.cc:1918] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:01:57.606186 29042 log.cc:1196] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/aa7ddc9bfa7342fe92896c83696c11ae
I20260928 03:01:57.606650 29042 ts_tablet_manager.cc:1939] T aa7ddc9bfa7342fe92896c83696c11ae P caa2934228e54d89b97ea95c8d5ec0ad: Deleting consensus metadata
I20260928 03:01:57.607956 28528 catalog_manager.cc:4841] TS a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549): tablet aa7ddc9bfa7342fe92896c83696c11ae (table test-table [id=0cb96dff028a417386a044bb0937a6cf]) successfully deleted
I20260928 03:01:57.608302 28528 catalog_manager.cc:4841] TS caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355): tablet aa7ddc9bfa7342fe92896c83696c11ae (table test-table [id=0cb96dff028a417386a044bb0937a6cf]) successfully deleted
I20260928 03:01:57.613802 29039 ts_tablet_manager.cc:1918] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:01:57.613981 29039 log.cc:1196] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/aa7ddc9bfa7342fe92896c83696c11ae
I20260928 03:01:57.614504 29039 ts_tablet_manager.cc:1939] T aa7ddc9bfa7342fe92896c83696c11ae P 1c8e82f9763c4e8f966b96eaf588fb2f: Deleting consensus metadata
I20260928 03:01:57.615520 29013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.580558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.12.182.65:39447)
0928 03:01:57.580729 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.580736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.590115 (+ 9379us) server_negotiation.cc:407] Connection header received
0928 03:01:57.591129 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.591136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.591227 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.591298 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:57.594335 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.595499 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.597888 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.598516 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.598618 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.606834 (+ 8216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:57.606849 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:57.606875 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:57.606927 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:57.611577 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:57.612504 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:57.612517 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:57.612520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:57.612703 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:57.613155 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:57.613160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:57.613164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:57.613717 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:57.614156 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.614877 (+ 721us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.615132 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 03:01:57.616119 28526 catalog_manager.cc:4841] TS 1c8e82f9763c4e8f966b96eaf588fb2f (127.12.182.65:39447): tablet aa7ddc9bfa7342fe92896c83696c11ae (table test-table [id=0cb96dff028a417386a044bb0937a6cf]) successfully deleted
I20260928 03:01:57.616897 28726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:57.713439 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 28603
I20260928 03:01:57.729401 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 28737
Sep 28 03:01:57 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 03:01:57.744176 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.730144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.12.182.66:37549)
0928 03:01:57.730358 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.730364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.730531 (+ 167us) server_negotiation.cc:407] Connection header received
0928 03:01:57.730678 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.730684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.730762 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.730860 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:57.732074 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.733099 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.734593 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.735178 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.735300 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.739187 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:57.739200 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:57.739215 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:57.739252 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:57.742000 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:57.742580 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:57.742586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:57.742589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:57.742702 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:57.742959 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:57.742961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:57.742963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:57.743307 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:57.743517 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.743781 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.743938 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:01:57.744762 28860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:57.831220 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 28737
I20260928 03:01:57.844841 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 28870
Sep 28 03:01:57 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 03:01:57.859056 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:01:57.845534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.12.182.67:38355)
0928 03:01:57.845733 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 03:01:57.845739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:01:57.845759 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:01:57.845926 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:01:57.845931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:01:57.846006 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:01:57.846090 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:01:57.847314 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.848023 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.849302 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:01:57.849814 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:01:57.849932 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:01:57.854207 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:01:57.854223 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:01:57.854238 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:01:57.854269 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:01:57.856666 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:57.857283 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:57.857289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:57.857292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:57.857420 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:01:57.857848 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:01:57.857851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:01:57.857853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:01:57.858186 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:01:57.858386 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 03:01:57.858651 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 03:01:57.858807 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 03:01:57.859623 28993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:57.947268 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 28870
I20260928 03:01:57.960107 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 28510
I20260928 03:01:57.961162 28572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:01:58.068277 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 28510
I20260928 03:02:18.086170 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:42827
--webserver_interface=127.12.182.126
--webserver_port=34939
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:42827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:02:18.401219 29047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:02:18.401502 29047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:02:18.401639 29047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:02:18.408844 29047 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 03:02:18.408946 29047 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 03:02:18.408988 29047 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 03:02:18.409060 29047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:02:18.409125 29047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:02:18.409220 29047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:02:18.409274 29047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:02:18.409356 29047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:02:18.419231 29047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:42827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:42827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=34939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:02:18.420488 29047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:02:18.421864 29047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:02:18.443264 29056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:18.443372 29053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:18.443346 29054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:02:18.444134 29047 server_base.cc:1034] running on GCE node
I20260928 03:02:18.445057 29047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:02:18.446648 29047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:02:18.447875 29047 hybrid_clock.cc:648] HybridClock initialized: now 1790564538447802 us; error 46 us; skew 500 ppm
I20260928 03:02:18.448244 29047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:02:18 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564538, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:02:18.457975 29047 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:02:18.459590 29047 webserver.cc:460] Webserver started at http://127.12.182.126:34939/ using document root <none> and password file <none>
I20260928 03:02:18.460345 29047 fs_manager.cc:359] Metadata directory not provided
I20260928 03:02:18.460459 29047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:02:18.466313 29047 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:02:18.469296 29063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:18.470669 29047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:02:18.470870 29047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "2f59068f840e4274b8661ca5781d9626"
format_stamp: "Formatted at 2026-09-28 03:01:55 on dist-test-slave-k5rg"
server_key: "a9b56fafb660ead950af051377b83a25"
server_key_iv: "82ba53398c19d41faaf8b7d0131ff574"
server_key_version: "encryptionkey@0"
I20260928 03:02:18.471859 29047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:02:18.488765 29047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:02:18.493062 29047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:02:18.493333 29047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:02:18.510934 29047 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:42827
I20260928 03:02:18.510957 29114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:42827 every 8 connection(s)
I20260928 03:02:18.512423 29047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:02:18.518844 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 29047
I20260928 03:02:18.519626 29115 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:02:18.519862 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:39447
--local_ip_for_outbound_sockets=127.12.182.65
--tserver_master_addrs=127.12.182.126:42827
--webserver_port=43583
--webserver_interface=127.12.182.65
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:02:18.524089 29115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Bootstrap starting.
I20260928 03:02:18.544269 29115 log.cc:824] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Log is configured to *not* fsync() on all Append() calls
I20260928 03:02:18.556345 29115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:02:18.556849 29115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Bootstrap complete.
I20260928 03:02:18.565670 29115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:02:18.566489 29115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f59068f840e4274b8661ca5781d9626, State: Initialized, Role: FOLLOWER
I20260928 03:02:18.567082 29115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:02:18.567325 29115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:02:18.567438 29115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:02:18.567593 29115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 1 FOLLOWER]: Advancing to term 2
I20260928 03:02:18.571424 29115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:02:18.571923 29115 leader_election.cc:302] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f59068f840e4274b8661ca5781d9626; no voters:
I20260928 03:02:18.572499 29115 leader_election.cc:288] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 03:02:18.572641 29119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 03:02:18.573714 29119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 2 LEADER]: Becoming Leader. State: Replica: 2f59068f840e4274b8661ca5781d9626, State: Running, Role: LEADER
I20260928 03:02:18.574290 29119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:02:18.574740 29115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:02:18.576768 29121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f59068f840e4274b8661ca5781d9626. Latest consensus state: current_term: 2 leader_uuid: "2f59068f840e4274b8661ca5781d9626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } } }
I20260928 03:02:18.577545 29121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: This master's current role is: LEADER
I20260928 03:02:18.577054 29120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2f59068f840e4274b8661ca5781d9626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } } }
I20260928 03:02:18.578241 29120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: This master's current role is: LEADER
I20260928 03:02:18.588104 29125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:02:18.596606 29125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:02:18.598109 29125 catalog_manager.cc:1259] Loaded cluster ID: ecc95aeffb2a4e6e87820c0d3011c284
I20260928 03:02:18.598208 29125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:02:18.602404 29125 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:02:18.604023 29125 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Loaded TSK: 0
I20260928 03:02:18.641702 29125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Generated new TSK 1
I20260928 03:02:18.642154 29125 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 03:02:18.877442 29117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:02:18.877739 29117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:02:18.877885 29117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:02:18.884922 29117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:02:18.885031 29117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:02:18.885182 29117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:02:18.894654 29117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:39447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=43583
--tserver_master_addrs=127.12.182.126:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:02:18.895820 29117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:02:18.897025 29117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:02:18.910764 29142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:18.910741 29143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:18.911343 29145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:02:18.912060 29117 server_base.cc:1034] running on GCE node
I20260928 03:02:18.912606 29117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:02:18.913471 29117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:02:18.914662 29117 hybrid_clock.cc:648] HybridClock initialized: now 1790564538914602 us; error 53 us; skew 500 ppm
I20260928 03:02:18.914944 29117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:02:18 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564538, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:02:18.917989 29117 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:02:18.919413 29117 webserver.cc:460] Webserver started at http://127.12.182.65:43583/ using document root <none> and password file <none>
I20260928 03:02:18.920027 29117 fs_manager.cc:359] Metadata directory not provided
I20260928 03:02:18.920152 29117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:02:18.925277 29117 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:02:18.928436 29152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:18.930306 29117 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 03:02:18.930481 29117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "1c8e82f9763c4e8f966b96eaf588fb2f"
format_stamp: "Formatted at 2026-09-28 03:01:56 on dist-test-slave-k5rg"
server_key: "7dcf0b66051160627017da086ba8a3e1"
server_key_iv: "9e907f77bdc309eb2f31a9ece32088c5"
server_key_version: "encryptionkey@0"
I20260928 03:02:18.931777 29117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:02:18.975730 29117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:02:18.980757 29117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:02:18.981042 29117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:02:18.982158 29117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:02:18.983728 29117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:02:18.983834 29117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:18.983933 29117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:02:18.983994 29117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:19.014555 29117 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:39447
I20260928 03:02:19.014632 29264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:39447 every 8 connection(s)
I20260928 03:02:19.016084 29117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:02:19.024969 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 29117
I20260928 03:02:19.025723 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:37549
--local_ip_for_outbound_sockets=127.12.182.66
--tserver_master_addrs=127.12.182.126:42827
--webserver_port=42427
--webserver_interface=127.12.182.66
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:02:19 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564538, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:02:19.039299 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:19.018846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:55475 (local address 127.12.182.126:42827)
0928 03:02:19.019445 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:19.019462 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:19.019909 (+ 447us) server_negotiation.cc:407] Connection header received
0928 03:02:19.021906 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:19.021919 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:19.022252 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:19.022575 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:19.023909 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.025131 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.027303 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.027950 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.028085 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:19.032178 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:19.032188 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:19.032206 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:19.032255 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:19.035138 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:19.035944 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:19.035950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:19.035955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:19.036106 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:19.036554 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:19.036557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:19.036559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:19.037043 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:19.037209 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:19.038049 (+ 840us) server_negotiation.cc:299] Negotiation successful
0928 03:02:19.038389 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":225,"threads_started":1}
I20260928 03:02:19.039525 29268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:19.019013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:55475)
0928 03:02:19.019748 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:19.019782 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 03:02:19.021398 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:02:19.022697 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:02:19.022705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:02:19.023223 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:02:19.023738 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:19.023755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.025319 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.025325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:19.027153 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:19.027167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.028168 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.028175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:19.028430 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:19.029144 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:02:19.029171 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:02:19.031851 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:02:19.035280 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:02:19.035289 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:02:19.035311 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:02:19.035829 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:02:19.036200 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:02:19.036205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:02:19.036208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:02:19.036464 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:02:19.037178 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:02:19.037186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:02:19.037563 (+ 377us) client_negotiation.cc:769] Sending connection context
0928 03:02:19.037886 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 03:02:19.038401 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":303,"threads_started":1}
I20260928 03:02:19.042569 29265 heartbeater.cc:347] Connected to a master server at 127.12.182.126:42827
I20260928 03:02:19.042860 29265 heartbeater.cc:464] Registering TS with master...
I20260928 03:02:19.043529 29265 heartbeater.cc:511] Master 127.12.182.126:42827 requested a full tablet report, sending...
I20260928 03:02:19.046103 29080 ts_manager.cc:194] Registered new tserver with Master: 1c8e82f9763c4e8f966b96eaf588fb2f (127.12.182.65:39447)
I20260928 03:02:19.048132 29080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:55475
W20260928 03:02:19.343231 29269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:02:19.343518 29269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:02:19.343619 29269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:02:19.350776 29269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:02:19.350870 29269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:02:19.350982 29269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:02:19.360351 29269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:37549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=42427
--tserver_master_addrs=127.12.182.126:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:02:19.361538 29269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:02:19.362800 29269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:02:19.374364 29274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:19.374918 29275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:19.375845 29277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:02:19.376533 29269 server_base.cc:1034] running on GCE node
I20260928 03:02:19.377074 29269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:02:19.378007 29269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:02:19.379211 29269 hybrid_clock.cc:648] HybridClock initialized: now 1790564539379138 us; error 54 us; skew 500 ppm
I20260928 03:02:19.379498 29269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:02:19 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564539, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:02:19.382352 29269 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:02:19.383582 29269 webserver.cc:460] Webserver started at http://127.12.182.66:42427/ using document root <none> and password file <none>
I20260928 03:02:19.384228 29269 fs_manager.cc:359] Metadata directory not provided
I20260928 03:02:19.384331 29269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:02:19.389370 29269 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:02:19.391975 29284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:19.393203 29269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 03:02:19.393342 29269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "a2618ef5787e4758bbbb4adacc63686c"
format_stamp: "Formatted at 2026-09-28 03:01:56 on dist-test-slave-k5rg"
server_key: "be6c5f6dbf163832b2dc4e04fc85cd15"
server_key_iv: "4e1bedd631ef2637c335f2fb5168824e"
server_key_version: "encryptionkey@0"
I20260928 03:02:19.394326 29269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:02:19.423840 29269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:02:19.427690 29269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:02:19.427932 29269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:02:19.428910 29269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:02:19.430217 29269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:02:19.430305 29269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:19.430403 29269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:02:19.430472 29269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:19.459100 29269 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:37549
I20260928 03:02:19.459156 29396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:37549 every 8 connection(s)
I20260928 03:02:19.460429 29269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:02:19.460776 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 29269
I20260928 03:02:19.461485 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:38355
--local_ip_for_outbound_sockets=127.12.182.67
--tserver_master_addrs=127.12.182.126:42827
--webserver_port=44485
--webserver_interface=127.12.182.67
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:02:19 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564539, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:02:19.490209 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:19.463026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:37901 (local address 127.12.182.126:42827)
0928 03:02:19.463198 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:19.463202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:19.464528 (+ 1326us) server_negotiation.cc:407] Connection header received
0928 03:02:19.466492 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:19.466497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:19.466577 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:19.466661 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:19.468114 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.468805 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.470349 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.470769 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.470887 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:19.474241 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:19.474246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:19.474250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:19.474284 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:19.486089 (+ 11805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:19.486991 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:19.486997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:19.486999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:19.487142 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:19.487622 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:19.487626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:19.487628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:19.488106 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:19.488252 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:19.489253 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0928 03:02:19.489865 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 03:02:19.491612 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:19.463621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:37901)
0928 03:02:19.464363 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:19.464399 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 03:02:19.466226 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:02:19.466925 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:02:19.466934 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:02:19.467413 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:02:19.467951 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:19.467968 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.468944 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.468948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:19.470201 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:19.470213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.470981 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.470988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:19.471203 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:19.471878 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:02:19.471901 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:02:19.474008 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:02:19.486296 (+ 12288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:02:19.486304 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:02:19.486320 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:02:19.486828 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:02:19.487263 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:02:19.487275 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:02:19.487277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:02:19.487481 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:02:19.488262 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:02:19.488295 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:02:19.488812 (+ 517us) client_negotiation.cc:769] Sending connection context
0928 03:02:19.489124 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 03:02:19.490141 (+ 1017us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":300,"threads_started":1}
I20260928 03:02:19.493208 29397 heartbeater.cc:347] Connected to a master server at 127.12.182.126:42827
I20260928 03:02:19.493503 29397 heartbeater.cc:464] Registering TS with master...
I20260928 03:02:19.494225 29397 heartbeater.cc:511] Master 127.12.182.126:42827 requested a full tablet report, sending...
I20260928 03:02:19.495736 29080 ts_manager.cc:194] Registered new tserver with Master: a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549)
I20260928 03:02:19.498571 29080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:37901
W20260928 03:02:19.852887 29400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:02:19.853132 29400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:02:19.853220 29400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:02:19.860275 29400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:02:19.860376 29400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:02:19.860499 29400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:02:19.869952 29400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:38355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=44485
--tserver_master_addrs=127.12.182.126:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:02:19.871127 29400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:02:19.872354 29400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:02:19.884841 29408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:19.884826 29405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:19.884876 29406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:02:19.885371 29400 server_base.cc:1034] running on GCE node
I20260928 03:02:19.885998 29400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:02:19.886945 29400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:02:19.888140 29400 hybrid_clock.cc:648] HybridClock initialized: now 1790564539888090 us; error 44 us; skew 500 ppm
I20260928 03:02:19.888446 29400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:02:19 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564539, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:02:19.891505 29400 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:02:19.892935 29400 webserver.cc:460] Webserver started at http://127.12.182.67:44485/ using document root <none> and password file <none>
I20260928 03:02:19.893555 29400 fs_manager.cc:359] Metadata directory not provided
I20260928 03:02:19.893705 29400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:02:19.898975 29400 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 03:02:19.901592 29415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:19.902998 29400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:02:19.903143 29400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "caa2934228e54d89b97ea95c8d5ec0ad"
format_stamp: "Formatted at 2026-09-28 03:01:57 on dist-test-slave-k5rg"
server_key: "11a6b09f038d00aa1c037663610aed88"
server_key_iv: "daf335752013ec22efae2ba3e054094a"
server_key_version: "encryptionkey@0"
I20260928 03:02:19.904101 29400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:02:19.917425 29400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:02:19.921525 29400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:02:19.921806 29400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:02:19.922784 29400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:02:19.924072 29400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:02:19.924171 29400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:19.924279 29400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:02:19.924347 29400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:19.954895 29400 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:38355
I20260928 03:02:19.954949 29527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:38355 every 8 connection(s)
I20260928 03:02:19.956269 29400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:02:19.963611 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 29400
Sep 28 03:02:19 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564539, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:02:19.976867 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:19.958730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:44511 (local address 127.12.182.126:42827)
0928 03:02:19.958949 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:19.958955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:19.960018 (+ 1063us) server_negotiation.cc:407] Connection header received
0928 03:02:19.961738 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:19.961744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:19.961816 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:19.961902 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:19.963266 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.964053 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.965923 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.966363 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.966467 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:19.970031 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:19.970036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:19.970039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:19.970078 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:19.973280 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:19.974183 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:19.974187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:19.974190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:19.974325 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:19.974899 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:19.974904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:19.974908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:19.975224 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:19.975363 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:19.976321 (+ 958us) server_negotiation.cc:299] Negotiation successful
0928 03:02:19.976487 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:02:19.977880 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:19.959190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:44511)
0928 03:02:19.959870 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:19.959906 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 03:02:19.961477 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:02:19.962102 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:02:19.962110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:02:19.962589 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:02:19.963113 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:19.963129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.964193 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.964197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:19.965787 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:19.965810 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.966567 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.966574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:19.966809 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:19.967434 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:02:19.967462 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:02:19.969834 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:02:19.973414 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:02:19.973423 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:02:19.973440 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:02:19.974056 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:02:19.974455 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:02:19.974460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:02:19.974464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:02:19.974695 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:02:19.975375 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:02:19.975382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:02:19.975846 (+ 464us) client_negotiation.cc:769] Sending connection context
0928 03:02:19.976185 (+ 339us) client_negotiation.cc:240] Negotiation successful
0928 03:02:19.976683 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":258,"threads_started":1}
I20260928 03:02:19.979152 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:19.964475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.12.182.126:42827)
0928 03:02:19.964918 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:19.964923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:19.965223 (+ 300us) server_negotiation.cc:407] Connection header received
0928 03:02:19.965366 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:19.965372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:19.965447 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:19.965552 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:19.966993 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.968011 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.969367 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.969999 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.970087 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:19.973254 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:19.973259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:19.973263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:19.973313 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:19.976437 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:19.977263 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:19.977267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:19.977269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:19.977374 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:19.977756 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:19.977761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:19.977764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:19.978087 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:19.978332 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:19.978594 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 03:02:19.978800 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":213,"threads_started":1}
I20260928 03:02:19.979313 29528 heartbeater.cc:347] Connected to a master server at 127.12.182.126:42827
I20260928 03:02:19.979568 29528 heartbeater.cc:464] Registering TS with master...
I20260928 03:02:19.980202 29528 heartbeater.cc:511] Master 127.12.182.126:42827 requested a full tablet report, sending...
I20260928 03:02:19.981660 29080 ts_manager.cc:194] Registered new tserver with Master: caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355)
I20260928 03:02:19.982668 29080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:44511
I20260928 03:02:19.983445 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:02:19.989841 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:19.984383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.12.182.126:42827)
0928 03:02:19.984540 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:19.984544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:19.985008 (+ 464us) server_negotiation.cc:407] Connection header received
0928 03:02:19.985210 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:19.985213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:19.985335 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:19.985453 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:02:19.985914 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.986571 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.987984 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.988417 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.988527 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:19.989047 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:02:19.989228 (+ 181us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 03:02:19.989558 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43276: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 03:02:19.989951 29531 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43276: token expired
W20260928 03:02:19.990474 29002 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.182.126:42827)
I20260928 03:02:20.002748 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:19.990958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.12.182.126:42827)
0928 03:02:19.991105 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:19.991110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:19.991125 (+ 15us) server_negotiation.cc:407] Connection header received
0928 03:02:19.991208 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:19.991211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:19.991285 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:19.991356 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:19.992616 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.993307 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.994690 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:19.995080 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:19.995171 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:19.997843 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:19.997848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:19.997851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:19.997881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:20.000310 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:20.000905 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:20.000909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:20.000911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:20.001026 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:20.001425 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:20.001428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:20.001431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:20.001777 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:20.001947 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:20.002273 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 03:02:20.002440 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 03:02:20.006253 29080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:02:20.013303 29080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:02:20.047066 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.024580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:38355 (local address 127.0.0.1:44228)
0928 03:02:20.038062 (+ 13482us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.038109 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 03:02:20.038295 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:02:20.039077 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:02:20.039085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:02:20.039116 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:02:20.039524 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:20.039537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.041687 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.041749 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:20.044252 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:20.044271 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.046061 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.046069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:20.046262 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.046320 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 03:02:20.046444 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 03:02:20.046606 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13115,"mutex_wait_us":1,"thread_start_us":13753,"threads_started":1}
I20260928 03:02:20.047129 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.025467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.12.182.65:39447)
0928 03:02:20.037512 (+ 12045us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:20.037522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:20.038677 (+ 1155us) server_negotiation.cc:407] Connection header received
0928 03:02:20.038949 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:20.038957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:20.039147 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:20.039324 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:02:20.040113 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.042135 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.044555 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.046048 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.046173 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.046390 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:20.046694 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 03:02:20.046852 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11849,"thread_start_us":11701,"threads_started":1}
I20260928 03:02:20.047456 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.025162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.0.0.1:34284)
0928 03:02:20.038581 (+ 13419us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.038597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:02:20.038774 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:02:20.039502 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:02:20.039506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:02:20.039525 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:02:20.039935 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:20.039947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.042324 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.042329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:20.044393 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:20.044406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.046339 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.046345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:20.046504 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.046537 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 03:02:20.046631 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 03:02:20.046771 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13313,"mutex_wait_us":2,"thread_start_us":12372,"threads_started":1}
I20260928 03:02:20.047963 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.024808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.12.182.67:38355)
0928 03:02:20.038062 (+ 13254us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:20.038070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:20.038503 (+ 433us) server_negotiation.cc:407] Connection header received
0928 03:02:20.038601 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:20.038609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:20.038860 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:20.039048 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:02:20.039746 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.041467 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.044434 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.045808 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.045946 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.046173 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:20.047493 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0928 03:02:20.047652 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13068,"thread_start_us":12722,"threads_started":1}
I20260928 03:02:20.051425 29265 heartbeater.cc:503] Master 127.12.182.126:42827 was elected leader, sending a full tablet report...
I20260928 03:02:20.051671 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.038699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:37549 (local address 127.0.0.1:47622)
0928 03:02:20.044765 (+ 6066us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.044781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:02:20.044889 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:02:20.045576 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:02:20.045581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:02:20.045628 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:02:20.045956 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:20.045968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.047805 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.047811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:20.049741 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:20.049754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.050983 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.050990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:20.051149 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.051171 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 03:02:20.051254 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 03:02:20.051369 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5940,"thread_start_us":5813,"threads_started":1}
I20260928 03:02:20.051616 29463 tablet_service.cc:1467] Processing CreateTablet for tablet b7451a3c835c4ac08e8dc0f5b09cc48d (DEFAULT_TABLE table=test-table [id=cfe67392ea8c47c5b5b35bdd6ad52634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:02:20.051625 29200 tablet_service.cc:1467] Processing CreateTablet for tablet b7451a3c835c4ac08e8dc0f5b09cc48d (DEFAULT_TABLE table=test-table [id=cfe67392ea8c47c5b5b35bdd6ad52634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:02:20.053064 29463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7451a3c835c4ac08e8dc0f5b09cc48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:02:20.053083 29200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7451a3c835c4ac08e8dc0f5b09cc48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:02:20.053632 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.038980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.12.182.66:37549)
0928 03:02:20.043109 (+ 4129us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:20.043115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:20.045003 (+ 1888us) server_negotiation.cc:407] Connection header received
0928 03:02:20.045098 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:20.045105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:20.045311 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:20.045481 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:02:20.046173 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.047656 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.049886 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.050775 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.050903 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.051081 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:20.053078 (+ 1997us) server_negotiation.cc:299] Negotiation successful
0928 03:02:20.053265 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3995,"thread_start_us":3867,"threads_started":1}
I20260928 03:02:20.056445 29332 tablet_service.cc:1467] Processing CreateTablet for tablet b7451a3c835c4ac08e8dc0f5b09cc48d (DEFAULT_TABLE table=test-table [id=cfe67392ea8c47c5b5b35bdd6ad52634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:02:20.057698 29332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7451a3c835c4ac08e8dc0f5b09cc48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:02:20.068554 29546 tablet_bootstrap.cc:492] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: Bootstrap starting.
I20260928 03:02:20.071642 29546 tablet_bootstrap.cc:654] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: Neither blocks nor log segments found. Creating new log.
I20260928 03:02:20.072474 29544 tablet_bootstrap.cc:492] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Bootstrap starting.
I20260928 03:02:20.072906 29546 log.cc:824] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:02:20.073387 29545 tablet_bootstrap.cc:492] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: Bootstrap starting.
I20260928 03:02:20.075976 29546 tablet_bootstrap.cc:492] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: No bootstrap required, opened a new log
I20260928 03:02:20.076265 29546 ts_tablet_manager.cc:1397] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 03:02:20.076927 29544 tablet_bootstrap.cc:654] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Neither blocks nor log segments found. Creating new log.
I20260928 03:02:20.078136 29544 log.cc:824] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Log is configured to *not* fsync() on all Append() calls
I20260928 03:02:20.080493 29545 tablet_bootstrap.cc:654] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: Neither blocks nor log segments found. Creating new log.
I20260928 03:02:20.081256 29544 tablet_bootstrap.cc:492] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: No bootstrap required, opened a new log
I20260928 03:02:20.081457 29545 log.cc:824] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: Log is configured to *not* fsync() on all Append() calls
I20260928 03:02:20.081507 29544 ts_tablet_manager.cc:1397] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 03:02:20.084592 29545 tablet_bootstrap.cc:492] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: No bootstrap required, opened a new log
I20260928 03:02:20.084823 29545 ts_tablet_manager.cc:1397] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260928 03:02:20.087069 29546 raft_consensus.cc:348] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:20.087505 29546 raft_consensus.cc:374] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:02:20.087599 29546 raft_consensus.cc:729] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2618ef5787e4758bbbb4adacc63686c, State: Initialized, Role: FOLLOWER
I20260928 03:02:20.088208 29546 consensus_queue.cc:260] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:20.089392 29397 heartbeater.cc:503] Master 127.12.182.126:42827 was elected leader, sending a full tablet report...
I20260928 03:02:20.091415 29546 ts_tablet_manager.cc:1428] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260928 03:02:20.091642 29544 raft_consensus.cc:348] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:20.092018 29544 raft_consensus.cc:374] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:02:20.092109 29544 raft_consensus.cc:729] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8e82f9763c4e8f966b96eaf588fb2f, State: Initialized, Role: FOLLOWER
I20260928 03:02:20.092645 29544 consensus_queue.cc:260] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:20.093770 29545 raft_consensus.cc:348] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:20.094170 29545 raft_consensus.cc:374] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:02:20.094271 29544 ts_tablet_manager.cc:1428] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260928 03:02:20.094283 29545 raft_consensus.cc:729] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa2934228e54d89b97ea95c8d5ec0ad, State: Initialized, Role: FOLLOWER
I20260928 03:02:20.094884 29545 consensus_queue.cc:260] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:20.095849 29528 heartbeater.cc:503] Master 127.12.182.126:42827 was elected leader, sending a full tablet report...
I20260928 03:02:20.097404 29545 ts_tablet_manager.cc:1428] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260928 03:02:20.148756 29550 raft_consensus.cc:482] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:02:20.149052 29550 raft_consensus.cc:504] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:20.150665 29550 leader_election.cc:288] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355), 1c8e82f9763c4e8f966b96eaf588fb2f (127.12.182.65:39447)
I20260928 03:02:20.157172 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.151116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:51893 (local address 127.12.182.65:39447)
0928 03:02:20.151322 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:20.151328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:20.152162 (+ 834us) server_negotiation.cc:407] Connection header received
0928 03:02:20.152339 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:20.152342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:20.152400 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:20.152472 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:02:20.153057 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.153799 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.155513 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.156277 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.156421 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.156505 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:20.156829 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 03:02:20.156948 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 03:02:20.157214 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.151214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:40101 (local address 127.12.182.67:38355)
0928 03:02:20.151422 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:20.151427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:20.152084 (+ 657us) server_negotiation.cc:407] Connection header received
0928 03:02:20.152273 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:20.152279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:20.152338 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:20.152410 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:02:20.152960 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.153707 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.155445 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.156292 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.156433 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.156505 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:20.156840 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 03:02:20.156961 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 03:02:20.157379 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.151189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.12.182.66:51893)
0928 03:02:20.152078 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.152092 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 03:02:20.152200 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:02:20.152495 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:02:20.152499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:02:20.152525 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:02:20.152917 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:20.152925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.153919 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.153923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:20.155365 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:20.155377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.156508 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.156515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:20.156667 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.156704 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 03:02:20.156779 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 03:02:20.156908 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":4,"thread_start_us":408,"threads_started":1}
I20260928 03:02:20.157402 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.151164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:38355 (local address 127.12.182.66:40101)
0928 03:02:20.151983 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.151999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:02:20.152148 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:02:20.152442 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:02:20.152446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:02:20.152469 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:02:20.152826 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:20.152835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.153832 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.153836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:20.155295 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:20.155307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.156527 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.156534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:20.156677 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.156724 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 03:02:20.156800 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 03:02:20.156927 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":62,"spinlock_wait_cycles":43008,"thread_start_us":482,"threads_started":1}
I20260928 03:02:20.158442 29220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7451a3c835c4ac08e8dc0f5b09cc48d" candidate_uuid: "a2618ef5787e4758bbbb4adacc63686c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" is_pre_election: true
I20260928 03:02:20.158625 29483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7451a3c835c4ac08e8dc0f5b09cc48d" candidate_uuid: "a2618ef5787e4758bbbb4adacc63686c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" is_pre_election: true
I20260928 03:02:20.158908 29220 raft_consensus.cc:2454] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2618ef5787e4758bbbb4adacc63686c in term 0.
I20260928 03:02:20.159085 29483 raft_consensus.cc:2454] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2618ef5787e4758bbbb4adacc63686c in term 0.
I20260928 03:02:20.159544 29285 leader_election.cc:302] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8e82f9763c4e8f966b96eaf588fb2f, a2618ef5787e4758bbbb4adacc63686c; no voters:
I20260928 03:02:20.159914 29550 raft_consensus.cc:2781] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:02:20.160037 29550 raft_consensus.cc:482] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:02:20.160135 29550 raft_consensus.cc:3037] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:02:20.162299 29550 raft_consensus.cc:504] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:20.162870 29550 leader_election.cc:288] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [CANDIDATE]: Term 1 election: Requested vote from peers caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355), 1c8e82f9763c4e8f966b96eaf588fb2f (127.12.182.65:39447)
I20260928 03:02:20.163283 29483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7451a3c835c4ac08e8dc0f5b09cc48d" candidate_uuid: "a2618ef5787e4758bbbb4adacc63686c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa2934228e54d89b97ea95c8d5ec0ad"
I20260928 03:02:20.163375 29220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7451a3c835c4ac08e8dc0f5b09cc48d" candidate_uuid: "a2618ef5787e4758bbbb4adacc63686c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f"
I20260928 03:02:20.163486 29483 raft_consensus.cc:3037] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:02:20.163564 29220 raft_consensus.cc:3037] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:02:20.166683 29220 raft_consensus.cc:2454] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2618ef5787e4758bbbb4adacc63686c in term 1.
I20260928 03:02:20.166683 29483 raft_consensus.cc:2454] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2618ef5787e4758bbbb4adacc63686c in term 1.
I20260928 03:02:20.167054 29285 leader_election.cc:302] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8e82f9763c4e8f966b96eaf588fb2f, a2618ef5787e4758bbbb4adacc63686c; no voters:
I20260928 03:02:20.167326 29550 raft_consensus.cc:2781] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:02:20.167802 29550 raft_consensus.cc:686] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 1 LEADER]: Becoming Leader. State: Replica: a2618ef5787e4758bbbb4adacc63686c, State: Running, Role: LEADER
I20260928 03:02:20.168249 29550 consensus_queue.cc:237] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:20.172880 29080 catalog_manager.cc:5495] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2618ef5787e4758bbbb4adacc63686c (127.12.182.66). New cstate: current_term: 1 leader_uuid: "a2618ef5787e4758bbbb4adacc63686c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } health_report { overall_health: UNKNOWN } } }
I20260928 03:02:20.207612 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.202742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.12.182.66:37549)
0928 03:02:20.202964 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:20.202968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:20.202985 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:02:20.203158 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:20.203162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:20.203239 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:20.203360 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:02:20.203636 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.204342 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.205714 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.206105 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.206191 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.206621 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:02:20.207097 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:02:20.207212 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:20.207289 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 03:02:20.207353 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 03:02:20.207692 29529 tablet.cc:2369] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:02:20.212384 29398 tablet.cc:2369] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:02:20.215026 29483 raft_consensus.cc:1261] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad [term 1 FOLLOWER]: Refusing update from remote peer a2618ef5787e4758bbbb4adacc63686c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:02:20.215034 29220 raft_consensus.cc:1261] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Refusing update from remote peer a2618ef5787e4758bbbb4adacc63686c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:02:20.215821 29555 consensus_queue.cc:1035] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [LEADER]: Connected to new peer: Peer: permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:02:20.216151 29550 consensus_queue.cc:1035] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:02:20.225785 29558 mvcc.cc:205] Tried to move back new op lower bound from 7334152356712034304 to 7334152356532842496. Current Snapshot: MvccSnapshot[applied={T|T < 7334152356712034304}]
I20260928 03:02:20.239516 29079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:02:20.239794 29079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:02:20.243026 29079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Sending DeleteTablet for 3 replicas of tablet b7451a3c835c4ac08e8dc0f5b09cc48d
I20260928 03:02:20.244236 29332 tablet_service.cc:1514] Processing DeleteTablet for tablet b7451a3c835c4ac08e8dc0f5b09cc48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:02:20 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:47622
I20260928 03:02:20.244230 29200 tablet_service.cc:1514] Processing DeleteTablet for tablet b7451a3c835c4ac08e8dc0f5b09cc48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:02:20 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:34284
I20260928 03:02:20.244252 29463 tablet_service.cc:1514] Processing DeleteTablet for tablet b7451a3c835c4ac08e8dc0f5b09cc48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:02:20 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:44228
I20260928 03:02:20.248126 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 29117
I20260928 03:02:20.265362 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.249031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.12.182.65:39447)
0928 03:02:20.249215 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:20.249221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:20.249243 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:02:20.249393 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:20.249399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:20.249477 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:20.249591 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:20.251355 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.252523 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.254398 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.255026 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.255171 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.258215 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:20.258228 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:20.258248 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:20.258293 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:20.262363 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:20.263147 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:20.263157 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:20.263161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:20.263322 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:20.263671 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:20.263693 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:20.263697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:20.264162 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:20.264494 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:20.264704 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 03:02:20.264945 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:02:20.266598 29260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:02:20.266712 29569 tablet_replica.cc:330] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: stopping tablet replica
I20260928 03:02:20.267182 29569 raft_consensus.cc:2229] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:02:20.267944 29569 raft_consensus.cc:2258] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:02:20.269850 29570 tablet_replica.cc:330] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: stopping tablet replica
I20260928 03:02:20.270270 29570 raft_consensus.cc:2229] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:02:20.270773 29570 raft_consensus.cc:2258] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:02:20.273231 29570 ts_tablet_manager.cc:1905] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:02:20.276872 29569 ts_tablet_manager.cc:1905] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:02:20.284101 29569 ts_tablet_manager.cc:1918] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:02:20.284277 29569 log.cc:1196] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/b7451a3c835c4ac08e8dc0f5b09cc48d
I20260928 03:02:20.284821 29569 ts_tablet_manager.cc:1939] T b7451a3c835c4ac08e8dc0f5b09cc48d P a2618ef5787e4758bbbb4adacc63686c: Deleting consensus metadata
I20260928 03:02:20.292232 29570 ts_tablet_manager.cc:1918] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:02:20.292431 29570 log.cc:1196] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/b7451a3c835c4ac08e8dc0f5b09cc48d
I20260928 03:02:20.292969 29570 ts_tablet_manager.cc:1939] T b7451a3c835c4ac08e8dc0f5b09cc48d P caa2934228e54d89b97ea95c8d5ec0ad: Deleting consensus metadata
I20260928 03:02:20.294804 29066 catalog_manager.cc:4841] TS caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355): tablet b7451a3c835c4ac08e8dc0f5b09cc48d (table test-table [id=cfe67392ea8c47c5b5b35bdd6ad52634]) successfully deleted
I20260928 03:02:20.295087 29066 catalog_manager.cc:4841] TS a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549): tablet b7451a3c835c4ac08e8dc0f5b09cc48d (table test-table [id=cfe67392ea8c47c5b5b35bdd6ad52634]) successfully deleted
I20260928 03:02:20.380352 29568 tablet_replica.cc:330] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: stopping tablet replica
I20260928 03:02:20.380553 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 29117
W20260928 03:02:20.395008 29064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.182.65:39447 (error 108)
I20260928 03:02:20.395192 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 29269
I20260928 03:02:20.397131 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.396588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.0.0.1:34302)
0928 03:02:20.396822 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.396948 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260928 03:02:20.397377 29064 catalog_manager.cc:4584] TS 1c8e82f9763c4e8f966b96eaf588fb2f (127.12.182.65:39447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7451a3c835c4ac08e8dc0f5b09cc48d: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39447: connect: Connection refused (error 111)
I20260928 03:02:20.411012 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.395930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.12.182.66:37549)
0928 03:02:20.396132 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:20.396138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:20.396230 (+ 92us) server_negotiation.cc:407] Connection header received
0928 03:02:20.396448 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:20.396453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:20.396536 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:20.396653 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:20.398381 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.399200 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.401073 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.401496 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.401587 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.404893 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:20.404906 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:20.404921 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:20.404956 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:20.407863 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:20.408726 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:20.408733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:20.408735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:20.408909 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:20.409321 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:20.409325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:20.409327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:20.409891 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:20.410185 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:20.410480 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 03:02:20.410648 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 03:02:20.411609 29392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:02:20.450966 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.450422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.0.0.1:34318)
0928 03:02:20.450646 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.450767 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 03:02:20.452057 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.451665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.0.0.1:34334)
0928 03:02:20.451826 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.451911 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 03:02:20.512777 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.512249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.0.0.1:34338)
0928 03:02:20.512483 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.512592 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 03:02:20.513947 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.513522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.0.0.1:34350)
0928 03:02:20.513720 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.513807 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 03:02:20.525492 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 29269
I20260928 03:02:20.540851 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 29400
I20260928 03:02:20.554139 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.541663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.12.182.67:38355)
0928 03:02:20.541845 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:20.541849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:20.541868 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:02:20.541981 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:20.541985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:20.542057 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:20.542149 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:20.543420 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.544171 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.545423 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:20.545883 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:20.545985 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:20.548665 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:20.548679 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:20.548695 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:20.548728 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:20.551444 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:20.552169 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:20.552177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:20.552179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:20.552322 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:20.552773 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:20.552777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:20.552779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:20.553138 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:20.553381 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:20.553670 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 03:02:20.553843 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:02:20.554705 29523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:02:20.595192 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.594729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.0.0.1:34354)
0928 03:02:20.594948 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.595040 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 03:02:20.596158 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.595789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.0.0.1:34358)
0928 03:02:20.595977 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.596047 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 03:02:20.641896 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 29400
I20260928 03:02:20.655196 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 29047
I20260928 03:02:20.656261 29110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:02:20.769435 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:20.768589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39447 (local address 127.0.0.1:34366)
0928 03:02:20.768805 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:20.768925 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 03:02:20.769474 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 29047
I20260928 03:02:40.787358 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:42827
--webserver_interface=127.12.182.126
--webserver_port=34939
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:42827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:02:41.131517 29576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:02:41.131803 29576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:02:41.131944 29576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:02:41.139055 29576 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 03:02:41.139168 29576 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 03:02:41.139223 29576 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 03:02:41.139293 29576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:02:41.139367 29576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:02:41.139462 29576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:02:41.139521 29576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:02:41.139595 29576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:02:41.149897 29576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:42827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:42827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=34939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:02:41.151134 29576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:02:41.152392 29576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:02:41.163582 29581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:41.163538 29582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:41.163604 29584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:02:41.164354 29576 server_base.cc:1034] running on GCE node
I20260928 03:02:41.165031 29576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:02:41.166453 29576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:02:41.167724 29576 hybrid_clock.cc:648] HybridClock initialized: now 1790564561167625 us; error 36 us; skew 500 ppm
I20260928 03:02:41.168049 29576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:02:41 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564561, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:02:41.171120 29576 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:02:41.172573 29576 webserver.cc:460] Webserver started at http://127.12.182.126:34939/ using document root <none> and password file <none>
I20260928 03:02:41.173225 29576 fs_manager.cc:359] Metadata directory not provided
I20260928 03:02:41.173309 29576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:02:41.178576 29576 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:02:41.181289 29591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:41.182698 29576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:02:41.182862 29576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "2f59068f840e4274b8661ca5781d9626"
format_stamp: "Formatted at 2026-09-28 03:01:55 on dist-test-slave-k5rg"
server_key: "a9b56fafb660ead950af051377b83a25"
server_key_iv: "82ba53398c19d41faaf8b7d0131ff574"
server_key_version: "encryptionkey@0"
I20260928 03:02:41.183830 29576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:02:41.197822 29576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:02:41.201680 29576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:02:41.201946 29576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:02:41.219426 29576 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:42827
I20260928 03:02:41.219449 29642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:42827 every 8 connection(s)
I20260928 03:02:41.220883 29576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:02:41.222558 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 29576
I20260928 03:02:41.223560 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:39447
--local_ip_for_outbound_sockets=127.12.182.65
--tserver_master_addrs=127.12.182.126:42827
--webserver_port=43583
--webserver_interface=127.12.182.65
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:02:41.229714 29643 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:02:41.233665 29643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Bootstrap starting.
I20260928 03:02:41.253507 29643 log.cc:824] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Log is configured to *not* fsync() on all Append() calls
I20260928 03:02:41.268982 29643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:02:41.269491 29643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Bootstrap complete.
I20260928 03:02:41.278187 29643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:02:41.279045 29643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f59068f840e4274b8661ca5781d9626, State: Initialized, Role: FOLLOWER
I20260928 03:02:41.279620 29643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:02:41.279846 29643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:02:41.279953 29643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:02:41.280102 29643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 2 FOLLOWER]: Advancing to term 3
I20260928 03:02:41.283834 29643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:02:41.284279 29643 leader_election.cc:302] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f59068f840e4274b8661ca5781d9626; no voters:
I20260928 03:02:41.284881 29643 leader_election.cc:288] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 03:02:41.285024 29647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 03:02:41.286101 29647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [term 3 LEADER]: Becoming Leader. State: Replica: 2f59068f840e4274b8661ca5781d9626, State: Running, Role: LEADER
I20260928 03:02:41.286684 29647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } }
I20260928 03:02:41.286927 29643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:02:41.289136 29649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f59068f840e4274b8661ca5781d9626. Latest consensus state: current_term: 3 leader_uuid: "2f59068f840e4274b8661ca5781d9626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } } }
I20260928 03:02:41.289480 29649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: This master's current role is: LEADER
I20260928 03:02:41.289827 29648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2f59068f840e4274b8661ca5781d9626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f59068f840e4274b8661ca5781d9626" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42827 } } }
I20260928 03:02:41.290098 29648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626 [sys.catalog]: This master's current role is: LEADER
I20260928 03:02:41.298074 29653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:02:41.307185 29653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:02:41.308642 29653 catalog_manager.cc:1259] Loaded cluster ID: ecc95aeffb2a4e6e87820c0d3011c284
I20260928 03:02:41.308794 29653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:02:41.313025 29653 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:02:41.314993 29653 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Loaded TSK: 0
I20260928 03:02:41.315214 29653 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f59068f840e4274b8661ca5781d9626: Loaded TSK: 1
F20260928 03:02:41.316339 29653 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f694d2c09ad google::LogMessage::Fail() at ??:0
@ 0x7f694d2c388a google::LogMessage::SendToLog() at ??:0
@ 0x7f694d2c04de google::LogMessage::Flush() at ??:0
@ 0x7f694d2c1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f69510247d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f695101373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6962cef1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6962cee783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6962d7a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f6962d7a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f6962dc45e9 std::function<>::operator()() at ??:0
@ 0x7f6962cf2cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6962cf0c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6962d78610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6962d78439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f694ff0c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f694ff0f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f694ff0f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f694feeaeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f69531c46db start_thread at ??:0
@ 0x7f694c15a61f clone at ??:0
W20260928 03:02:41.614864 29645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:02:41.615123 29645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:02:41.615213 29645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:02:41.622265 29645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:02:41.622360 29645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:02:41.622483 29645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:02:41.631789 29645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:39447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=43583
--tserver_master_addrs=127.12.182.126:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:02:41.632921 29645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:02:41.634155 29645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:02:41.646060 29691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:41.646521 29693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:41.646572 29690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:02:41.646790 29645 server_base.cc:1034] running on GCE node
I20260928 03:02:41.647374 29645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:02:41.648204 29645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:02:41.649394 29645 hybrid_clock.cc:648] HybridClock initialized: now 1790564561649355 us; error 28 us; skew 500 ppm
I20260928 03:02:41.649701 29645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:02:41 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564561, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:02:41.652470 29645 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:02:41.653795 29645 webserver.cc:460] Webserver started at http://127.12.182.65:43583/ using document root <none> and password file <none>
I20260928 03:02:41.654395 29645 fs_manager.cc:359] Metadata directory not provided
I20260928 03:02:41.654507 29645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:02:41.659291 29645 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 03:02:41.661865 29700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:41.663013 29645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 03:02:41.663146 29645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "1c8e82f9763c4e8f966b96eaf588fb2f"
format_stamp: "Formatted at 2026-09-28 03:01:56 on dist-test-slave-k5rg"
server_key: "7dcf0b66051160627017da086ba8a3e1"
server_key_iv: "9e907f77bdc309eb2f31a9ece32088c5"
server_key_version: "encryptionkey@0"
I20260928 03:02:41.664037 29645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:02:41.679708 29645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:02:41.683269 29645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:02:41.683509 29645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:02:41.684409 29645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:02:41.686467 29707 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 03:02:41.692140 29645 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 03:02:41.692260 29645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260928 03:02:41.692378 29645 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 03:02:41.695587 29645 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 03:02:41.695665 29645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260928 03:02:41.696012 29707 tablet_bootstrap.cc:492] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Bootstrap starting.
I20260928 03:02:41.721678 29707 log.cc:824] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Log is configured to *not* fsync() on all Append() calls
I20260928 03:02:41.732553 29645 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:39447
I20260928 03:02:41.732587 29814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:39447 every 8 connection(s)
I20260928 03:02:41.733047 29707 tablet_bootstrap.cc:492] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:02:41.733543 29707 tablet_bootstrap.cc:492] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Bootstrap complete.
I20260928 03:02:41.733840 29645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:02:41.734815 29707 ts_tablet_manager.cc:1397] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.019s
I20260928 03:02:41.738780 29817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:41.736520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:46955)
0928 03:02:41.737437 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:41.737790 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":415,"threads_started":1}
I20260928 03:02:41.739301 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 29645
W20260928 03:02:41.739532 29815 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:42827 (0 consecutive failures): Network error: Failed to ping master at 127.12.182.126:42827: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
I20260928 03:02:41.740049 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:37549
--local_ip_for_outbound_sockets=127.12.182.66
--tserver_master_addrs=127.12.182.126:42827
--webserver_port=42427
--webserver_interface=127.12.182.66
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:02:41.740351 29817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:41.740015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:57509)
0928 03:02:41.740163 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:41.740233 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 03:02:41.741178 29817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:41.740842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:33455)
0928 03:02:41.740973 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:41.741050 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 03:02:41.741369 29815 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 03:02:41.743342 29707 raft_consensus.cc:348] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:41.744477 29707 raft_consensus.cc:729] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8e82f9763c4e8f966b96eaf588fb2f, State: Initialized, Role: FOLLOWER
I20260928 03:02:41.745106 29707 consensus_queue.cc:260] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:41.746595 29817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:41.746278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:52913)
0928 03:02:41.746422 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:41.746489 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 03:02:41.746724 29707 ts_tablet_manager.cc:1428] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20260928 03:02:41.747550 29817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:41.747251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:42655)
0928 03:02:41.747380 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:41.747440 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 03:02:42.058858 29819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:02:42.059144 29819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:02:42.059278 29819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:02:42.066557 29819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:02:42.066704 29819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:02:42.066886 29819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:02:42.076493 29819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:37549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=42427
--tserver_master_addrs=127.12.182.126:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:02:42.077790 29819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:02:42.079061 29819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:02:42.091543 29827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:42.091614 29825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:42.091650 29824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:02:42.092375 29819 server_base.cc:1034] running on GCE node
I20260928 03:02:42.092931 29819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:02:42.093848 29819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:02:42.095073 29819 hybrid_clock.cc:648] HybridClock initialized: now 1790564562095023 us; error 41 us; skew 500 ppm
I20260928 03:02:42.095386 29819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:02:42 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564562, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:02:42.098446 29819 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:02:42.099828 29819 webserver.cc:460] Webserver started at http://127.12.182.66:42427/ using document root <none> and password file <none>
I20260928 03:02:42.100493 29819 fs_manager.cc:359] Metadata directory not provided
I20260928 03:02:42.100611 29819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:02:42.105890 29819 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:02:42.108536 29834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:42.109836 29819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:02:42.110003 29819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "a2618ef5787e4758bbbb4adacc63686c"
format_stamp: "Formatted at 2026-09-28 03:01:56 on dist-test-slave-k5rg"
server_key: "be6c5f6dbf163832b2dc4e04fc85cd15"
server_key_iv: "4e1bedd631ef2637c335f2fb5168824e"
server_key_version: "encryptionkey@0"
I20260928 03:02:42.111035 29819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:02:42.134037 29819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:02:42.137830 29819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:02:42.138077 29819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:02:42.139026 29819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:02:42.140350 29819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:02:42.140445 29819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:42.140552 29819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:02:42.140622 29819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:42.170449 29819 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:37549
I20260928 03:02:42.170535 29946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:37549 every 8 connection(s)
I20260928 03:02:42.171892 29819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:02:42.174242 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 29819
I20260928 03:02:42.175093 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:38355
--local_ip_for_outbound_sockets=127.12.182.67
--tserver_master_addrs=127.12.182.126:42827
--webserver_port=44485
--webserver_interface=127.12.182.67
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:02:42.177165 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:42.175297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:35491)
0928 03:02:42.176088 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:42.176348 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":313,"threads_started":1}
W20260928 03:02:42.177817 29947 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:42827 (0 consecutive failures): Network error: Failed to ping master at 127.12.182.126:42827: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
I20260928 03:02:42.178637 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:42.178318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:48199)
0928 03:02:42.178463 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:42.178533 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 03:02:42.179454 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:42.179126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:39351)
0928 03:02:42.179281 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:42.179352 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 03:02:42.179683 29947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 03:02:42.488492 29950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:02:42.488749 29950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:02:42.488870 29950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:02:42.496236 29950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:02:42.496337 29950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:02:42.496456 29950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:02:42.505942 29950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:38355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=44485
--tserver_master_addrs=127.12.182.126:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:02:42.507085 29950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:02:42.508275 29950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:02:42.520574 29956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:42.520645 29955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:42.520650 29958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:02:42.521324 29950 server_base.cc:1034] running on GCE node
I20260928 03:02:42.521904 29950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:02:42.522747 29950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:02:42.523933 29950 hybrid_clock.cc:648] HybridClock initialized: now 1790564562523889 us; error 34 us; skew 500 ppm
I20260928 03:02:42.524211 29950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:02:42 dist-test-slave-k5rg krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564562, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:02:42.527079 29950 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:02:42.528357 29950 webserver.cc:460] Webserver started at http://127.12.182.67:44485/ using document root <none> and password file <none>
I20260928 03:02:42.528963 29950 fs_manager.cc:359] Metadata directory not provided
I20260928 03:02:42.529069 29950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:02:42.534142 29950 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:02:42.536831 29965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:42.538108 29950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:02:42.538254 29950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "caa2934228e54d89b97ea95c8d5ec0ad"
format_stamp: "Formatted at 2026-09-28 03:01:57 on dist-test-slave-k5rg"
server_key: "11a6b09f038d00aa1c037663610aed88"
server_key_iv: "daf335752013ec22efae2ba3e054094a"
server_key_version: "encryptionkey@0"
I20260928 03:02:42.539184 29950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:02:42.562920 29950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:02:42.566879 29950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:02:42.567162 29950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:02:42.568161 29950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:02:42.569461 29950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:02:42.569552 29950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:42.569674 29950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:02:42.569747 29950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:42.600381 29950 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:38355
I20260928 03:02:42.600464 30077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:38355 every 8 connection(s)
I20260928 03:02:42.601805 29950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:02:42.606480 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:42.604656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:39267)
0928 03:02:42.605365 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:42.605642 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":262,"threads_started":1}
I20260928 03:02:42.607056 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 29950
W20260928 03:02:42.607187 30078 heartbeater.cc:644] Failed to heartbeat to 127.12.182.126:42827 (0 consecutive failures): Network error: Failed to ping master at 127.12.182.126:42827: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
I20260928 03:02:42.608119 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:42.607724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:36649)
0928 03:02:42.607911 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:42.607996 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 03:02:42.608951 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:42.608604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:55501)
0928 03:02:42.608770 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:42.608846 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 03:02:42.609163 30078 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 03:02:42.748986 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:42.748275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:48945)
0928 03:02:42.748711 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:42.748799 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":212,"threads_started":1}
I20260928 03:02:43.180907 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:43.180182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:56109)
0928 03:02:43.180591 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:43.180675 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":192,"threads_started":1}
I20260928 03:02:43.374217 30084 raft_consensus.cc:482] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:02:43.374500 30084 raft_consensus.cc:504] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:43.375958 30084 leader_election.cc:288] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355), a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549)
Sep 28 03:02:43 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564561, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
Sep 28 03:02:43 dist-test-slave-k5rg krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564561, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 03:02:43.395502 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:43.376217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:38355 (local address 127.12.182.65:33803)
0928 03:02:43.376743 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:43.376784 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 03:02:43.377373 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:02:43.379981 (+ 2608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:02:43.379992 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:02:43.380828 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:02:43.381520 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:43.381539 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:43.383009 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:43.383012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:43.384376 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:43.384388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:43.385327 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:43.385335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:43.385557 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:43.386351 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:02:43.386367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:02:43.389122 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:02:43.392369 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:02:43.392373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:02:43.392375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:02:43.392816 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:02:43.393260 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:02:43.393263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:02:43.393265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:02:43.393437 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:02:43.394119 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:02:43.394123 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:02:43.394336 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 03:02:43.394557 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 03:02:43.395027 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":179,"spinlock_wait_cycles":29568,"thread_start_us":270,"threads_started":1}
I20260928 03:02:43.395524 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:43.377346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:44335 (local address 127.12.182.66:37549)
0928 03:02:43.378264 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:43.378274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:43.378308 (+ 34us) server_negotiation.cc:407] Connection header received
0928 03:02:43.379730 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:43.379738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:43.380022 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:43.380290 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:43.381710 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:43.382842 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:43.384504 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:43.385091 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:43.385261 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:43.389015 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:43.389026 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:43.389044 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:43.389093 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:43.391865 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:43.392709 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:43.392717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:43.392725 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:43.392929 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:43.393368 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:43.393372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:43.393376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:43.393918 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:43.394055 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:43.394799 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 03:02:43.395173 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":529,"threads_started":1}
I20260928 03:02:43.395579 30087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:43.376964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:37549 (local address 127.12.182.65:44335)
0928 03:02:43.377505 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:43.377519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 03:02:43.377648 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:02:43.380275 (+ 2627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:02:43.380280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:02:43.380895 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:02:43.381519 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:43.381540 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:43.383007 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:43.383010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:43.384379 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:02:43.384391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:02:43.385275 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:02:43.385282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:02:43.385555 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:43.386298 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:02:43.386320 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:02:43.388653 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:02:43.391996 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:02:43.392005 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:02:43.392019 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:02:43.392574 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:02:43.393028 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:02:43.393033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:02:43.393036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:02:43.393260 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:02:43.394052 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:02:43.394059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:02:43.394338 (+ 279us) client_negotiation.cc:769] Sending connection context
0928 03:02:43.394557 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 03:02:43.395176 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":265,"spinlock_wait_cycles":22016,"thread_start_us":548,"threads_started":1}
I20260928 03:02:43.395555 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:43.376466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:33803 (local address 127.12.182.67:38355)
0928 03:02:43.376991 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:43.377005 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:43.377058 (+ 53us) server_negotiation.cc:407] Connection header received
0928 03:02:43.379200 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:43.379210 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:43.379554 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:43.379829 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:43.381709 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:43.382834 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:43.384506 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:43.385091 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:43.385257 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:43.389350 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:43.389357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:43.389368 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:43.389405 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:43.392248 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:43.392937 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:43.392942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:43.392947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:43.393140 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:43.393541 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:43.393544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:43.393546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:43.394000 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:43.394179 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:43.394809 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 03:02:43.395132 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":251,"threads_started":1}
W20260928 03:02:43.398464 29703 leader_election.cc:341] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355): Not found: Tablet not found: b7451a3c835c4ac08e8dc0f5b09cc48d
W20260928 03:02:43.398731 29703 leader_election.cc:341] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549): Not found: Tablet not found: b7451a3c835c4ac08e8dc0f5b09cc48d
I20260928 03:02:43.398837 29703 leader_election.cc:302] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c8e82f9763c4e8f966b96eaf588fb2f; no voters: a2618ef5787e4758bbbb4adacc63686c, caa2934228e54d89b97ea95c8d5ec0ad
I20260928 03:02:43.399147 30084 raft_consensus.cc:2726] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:02:43.610423 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:43.609638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:43431)
0928 03:02:43.610061 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:43.610207 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":197,"threads_started":1}
I20260928 03:02:43.750223 30087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:43.749799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:35727)
0928 03:02:43.749985 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:43.750078 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 03:02:44.182435 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:44.181753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:51813)
0928 03:02:44.182202 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:44.182281 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":248,"threads_started":1}
I20260928 03:02:44.611958 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:44.611227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:37675)
0928 03:02:44.611658 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:44.611757 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":223,"threads_started":1}
I20260928 03:02:44.751794 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:44.751046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:58037)
0928 03:02:44.751541 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:44.751624 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":222,"threads_started":1}
I20260928 03:02:45.183929 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:45.183217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:35985)
0928 03:02:45.183669 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:45.183750 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":231,"threads_started":1}
I20260928 03:02:45.256249 30094 raft_consensus.cc:482] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:02:45.256430 30094 raft_consensus.cc:504] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:45.257030 30094 leader_election.cc:288] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355), a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549)
W20260928 03:02:45.258049 29703 leader_election.cc:341] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355): Not found: Tablet not found: b7451a3c835c4ac08e8dc0f5b09cc48d
W20260928 03:02:45.258268 29703 leader_election.cc:341] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549): Not found: Tablet not found: b7451a3c835c4ac08e8dc0f5b09cc48d
I20260928 03:02:45.258354 29703 leader_election.cc:302] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c8e82f9763c4e8f966b96eaf588fb2f; no voters: a2618ef5787e4758bbbb4adacc63686c, caa2934228e54d89b97ea95c8d5ec0ad
I20260928 03:02:45.258630 30094 raft_consensus.cc:2726] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:02:45.613381 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:45.612717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:44109)
0928 03:02:45.613131 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:45.613210 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":218,"threads_started":1}
I20260928 03:02:45.753367 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:45.752616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:42327)
0928 03:02:45.753072 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:45.753189 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":216,"threads_started":1}
I20260928 03:02:46.185420 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:46.184707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:43643)
0928 03:02:46.185154 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:46.185238 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":213,"threads_started":1}
I20260928 03:02:46.614939 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:46.614232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:58543)
0928 03:02:46.614626 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:46.614724 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":196,"threads_started":1}
I20260928 03:02:46.755075 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:46.754382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:49709)
0928 03:02:46.754803 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:46.754905 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":229,"threads_started":1}
I20260928 03:02:47.187176 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:47.186396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:44897)
0928 03:02:47.186910 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:47.186994 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":297,"threads_started":1}
I20260928 03:02:47.616477 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:47.615787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:42869)
0928 03:02:47.616222 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:47.616308 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":218,"threads_started":1}
I20260928 03:02:47.756685 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:47.755957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:51873)
0928 03:02:47.756422 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:47.756505 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":245,"threads_started":1}
I20260928 03:02:48.082602 30103 raft_consensus.cc:482] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:02:48.082796 30103 raft_consensus.cc:504] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:48.083456 30103 leader_election.cc:288] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355), a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549)
W20260928 03:02:48.084417 29703 leader_election.cc:341] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355): Not found: Tablet not found: b7451a3c835c4ac08e8dc0f5b09cc48d
W20260928 03:02:48.084673 29703 leader_election.cc:341] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549): Not found: Tablet not found: b7451a3c835c4ac08e8dc0f5b09cc48d
I20260928 03:02:48.084779 29703 leader_election.cc:302] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c8e82f9763c4e8f966b96eaf588fb2f; no voters: a2618ef5787e4758bbbb4adacc63686c, caa2934228e54d89b97ea95c8d5ec0ad
I20260928 03:02:48.085001 30103 raft_consensus.cc:2726] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:02:48.188855 30104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:48.188093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:49239)
0928 03:02:48.188586 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:48.188673 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":261,"threads_started":1}
I20260928 03:02:48.618083 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:48.617302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:43985)
0928 03:02:48.617831 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:48.617912 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":250,"threads_started":1}
I20260928 03:02:48.758287 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:48.757524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:35271)
0928 03:02:48.757962 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:48.758076 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":233,"threads_started":1}
I20260928 03:02:49.190621 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:49.189827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:43431)
0928 03:02:49.190349 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:49.190441 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":238,"threads_started":1}
I20260928 03:02:49.620621 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:49.619605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:50987)
0928 03:02:49.620324 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:49.620422 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":407,"threads_started":1}
I20260928 03:02:49.759903 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:49.759184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:47523)
0928 03:02:49.759614 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:49.759735 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260928 03:02:50.192175 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:50.191502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:38407)
0928 03:02:50.191899 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:50.191988 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":222,"threads_started":1}
I20260928 03:02:50.622809 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:50.622144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:57803)
0928 03:02:50.622539 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:50.622623 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":201,"threads_started":1}
I20260928 03:02:50.761862 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:50.760974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:43195)
0928 03:02:50.761462 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:50.761568 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":247,"threads_started":1}
I20260928 03:02:51.193853 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:51.193082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:40423)
0928 03:02:51.193557 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:51.193690 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":227,"threads_started":1}
I20260928 03:02:51.619932 30114 raft_consensus.cc:482] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:02:51.620136 30114 raft_consensus.cc:504] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:51.620885 30114 leader_election.cc:288] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355), a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549)
W20260928 03:02:51.621734 29703 leader_election.cc:341] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355): Not found: Tablet not found: b7451a3c835c4ac08e8dc0f5b09cc48d
W20260928 03:02:51.621941 29703 leader_election.cc:341] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549): Not found: Tablet not found: b7451a3c835c4ac08e8dc0f5b09cc48d
I20260928 03:02:51.622025 29703 leader_election.cc:302] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c8e82f9763c4e8f966b96eaf588fb2f; no voters: a2618ef5787e4758bbbb4adacc63686c, caa2934228e54d89b97ea95c8d5ec0ad
I20260928 03:02:51.622255 30114 raft_consensus.cc:2726] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:02:51.624231 30115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:51.623562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:37457)
0928 03:02:51.624002 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:51.624090 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":230,"threads_started":1}
I20260928 03:02:51.763759 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:51.762887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:35977)
0928 03:02:51.763421 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:51.763519 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":259,"threads_started":1}
I20260928 03:02:52.195353 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:52.194664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:35839)
0928 03:02:52.195094 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:52.195173 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":232,"threads_started":1}
I20260928 03:02:52.625697 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:52.625018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:47061)
0928 03:02:52.625429 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:52.625506 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260928 03:02:52.765311 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:52.764636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:56847)
0928 03:02:52.765050 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:52.765133 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":207,"threads_started":1}
I20260928 03:02:53.196898 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:53.196204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:53629)
0928 03:02:53.196645 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:53.196729 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":203,"threads_started":1}
I20260928 03:02:53.627250 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:53.626505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:48575)
0928 03:02:53.627000 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:53.627084 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":258,"threads_started":1}
I20260928 03:02:53.766835 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:53.766132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:48041)
0928 03:02:53.766553 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:53.766630 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":226,"threads_started":1}
I20260928 03:02:54.198577 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:54.197845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:56483)
0928 03:02:54.198327 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:54.198402 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":250,"threads_started":1}
I20260928 03:02:54.628753 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:54.628083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:38893)
0928 03:02:54.628506 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:54.628586 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":223,"threads_started":1}
I20260928 03:02:54.768467 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:54.767732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:54629)
0928 03:02:54.768147 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:54.768261 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":210,"threads_started":1}
I20260928 03:02:54.937712 30126 raft_consensus.cc:482] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:02:54.937924 30126 raft_consensus.cc:504] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa2934228e54d89b97ea95c8d5ec0ad" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 38355 } } peers { permanent_uuid: "a2618ef5787e4758bbbb4adacc63686c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 37549 } } peers { permanent_uuid: "1c8e82f9763c4e8f966b96eaf588fb2f" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39447 } }
I20260928 03:02:54.938616 30126 leader_election.cc:288] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355), a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549)
W20260928 03:02:54.939528 29703 leader_election.cc:341] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2618ef5787e4758bbbb4adacc63686c (127.12.182.66:37549): Not found: Tablet not found: b7451a3c835c4ac08e8dc0f5b09cc48d
W20260928 03:02:54.939697 29703 leader_election.cc:341] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caa2934228e54d89b97ea95c8d5ec0ad (127.12.182.67:38355): Not found: Tablet not found: b7451a3c835c4ac08e8dc0f5b09cc48d
I20260928 03:02:54.939786 29703 leader_election.cc:302] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c8e82f9763c4e8f966b96eaf588fb2f; no voters: a2618ef5787e4758bbbb4adacc63686c, caa2934228e54d89b97ea95c8d5ec0ad
I20260928 03:02:54.939998 30126 raft_consensus.cc:2726] T b7451a3c835c4ac08e8dc0f5b09cc48d P 1c8e82f9763c4e8f966b96eaf588fb2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:02:55.200155 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:55.199421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:43371)
0928 03:02:55.199907 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:55.199995 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":226,"threads_started":1}
I20260928 03:02:55.630263 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:55.629565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:57293)
0928 03:02:55.630022 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:55.630110 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":228,"threads_started":1}
I20260928 03:02:55.769950 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:55.769247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:54679)
0928 03:02:55.769671 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:55.769752 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":213,"threads_started":1}
I20260928 03:02:56.201694 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:56.200952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:49089)
0928 03:02:56.201391 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:56.201470 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":221,"threads_started":1}
I20260928 03:02:56.631912 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:56.631168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:55823)
0928 03:02:56.631601 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:56.631697 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":211,"threads_started":1}
I20260928 03:02:56.771884 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:56.771096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.65:51643)
0928 03:02:56.771587 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:56.771682 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":241,"threads_started":1}
I20260928 03:02:57.203616 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:57.202864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.66:50369)
0928 03:02:57.203345 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:57.203452 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":227,"threads_started":1}
I20260928 03:02:57.617437 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 29645
I20260928 03:02:57.633888 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:57.633040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:42827 (local address 127.12.182.67:38071)
0928 03:02:57.633503 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 03:02:57.633665 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.126:42827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":221,"threads_started":1}
I20260928 03:02:57.638234 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:57.618565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52120 (local address 127.12.182.65:39447)
0928 03:02:57.619106 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:57.619116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:57.619149 (+ 33us) server_negotiation.cc:407] Connection header received
0928 03:02:57.619227 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:57.619236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:57.619520 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:57.619700 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:57.621526 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:57.622570 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:57.623828 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:57.624505 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:57.624602 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:57.632445 (+ 7843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:57.632462 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:57.632482 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:57.632522 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:57.635690 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:57.636364 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:57.636371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:57.636373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:57.636505 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:57.636805 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:57.636808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:57.636810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:57.637144 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:57.637392 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:57.637775 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 03:02:57.637980 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":496,"threads_started":1}
I20260928 03:02:57.639436 29810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:02:57.728916 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 29645
I20260928 03:02:57.743144 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 29819
I20260928 03:02:57.756003 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:57.743975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.12.182.66:37549)
0928 03:02:57.744600 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:57.744604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:57.744621 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:02:57.744684 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:57.744687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:57.744745 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:57.744837 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:57.746217 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:57.747066 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:57.748326 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:57.748748 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:57.748837 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:57.751244 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:57.751249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:57.751252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:57.751287 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:57.753753 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:57.754389 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:57.754392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:57.754395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:57.754518 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:57.754896 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:57.754899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:57.754901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:57.755141 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:57.755335 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:57.755546 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 03:02:57.755699 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":406,"threads_started":1}
I20260928 03:02:57.756650 29942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:02:57.839296 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 29819
I20260928 03:02:57.853879 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 29950
I20260928 03:02:57.867118 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:57.854601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44282 (local address 127.12.182.67:38355)
0928 03:02:57.855135 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:57.855140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:57.855156 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:02:57.855215 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:57.855218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:57.855300 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:57.855396 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:57.856810 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:57.857591 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:57.858939 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:57.859365 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:57.859474 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:57.862032 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:57.862037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:57.862040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:57.862076 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:57.864489 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:57.865147 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:57.865153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:57.865157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:57.865320 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:57.865708 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:57.865713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:57.865717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:57.865975 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:57.866246 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:57.866538 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 03:02:57.866751 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":265,"threads_started":1}
I20260928 03:02:57.867879 30073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:02:57.950665 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 29950
2026-09-28T03:02:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63856 ms)
[----------] 30 tests from SecurityITest (282837 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:02:58 dist-test-slave-k5rg krb5kdc[30143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:02:58 dist-test-slave-k5rg krb5kdc[30143](info): set up 2 sockets
Sep 28 03:02:58 dist-test-slave-k5rg krb5kdc[30143](info): commencing operation
krb5kdc: starting...
W20260928 03:02:58.973731 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:02:59 dist-test-slave-k5rg krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:02:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:02:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:02:59.207044 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:45675
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:02:59.525872 30156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:02:59.526145 30156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:02:59.526250 30156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:02:59.533499 30156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:02:59.533641 30156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:02:59.533691 30156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:02:59.533747 30156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:02:59.533790 30156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:02:59.543964 30156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:45675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:02:59.545264 30156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:02:59.546607 30156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:02:59.558211 30164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:59.558277 30162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:02:59.558259 30161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:02:59.558611 30156 server_base.cc:1034] running on GCE node
I20260928 03:02:59.559562 30156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:02:59.561122 30156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:02:59.562343 30156 hybrid_clock.cc:648] HybridClock initialized: now 1790564579562285 us; error 44 us; skew 500 ppm
I20260928 03:02:59.562685 30156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:02:59 dist-test-slave-k5rg krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564579, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:02:59.565807 30156 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:02:59.567260 30156 webserver.cc:460] Webserver started at http://127.12.182.126:46321/ using document root <none> and password file <none>
I20260928 03:02:59.567953 30156 fs_manager.cc:359] Metadata directory not provided
I20260928 03:02:59.568070 30156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:02:59.568360 30156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:02:59.571398 30156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "d70dba8053f94a17892284f07d33d931"
format_stamp: "Formatted at 2026-09-28 03:02:59 on dist-test-slave-k5rg"
server_key: "fc70effd4bc5b9fffe98cfa8043ff4e5"
server_key_iv: "257a971638968cdd5ebf27a4078988a4"
server_key_version: "encryptionkey@0"
I20260928 03:02:59.572139 30156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "d70dba8053f94a17892284f07d33d931"
format_stamp: "Formatted at 2026-09-28 03:02:59 on dist-test-slave-k5rg"
server_key: "fc70effd4bc5b9fffe98cfa8043ff4e5"
server_key_iv: "257a971638968cdd5ebf27a4078988a4"
server_key_version: "encryptionkey@0"
I20260928 03:02:59.577163 30156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:02:59.580751 30171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:02:59.582095 30156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:02:59.582266 30156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "d70dba8053f94a17892284f07d33d931"
format_stamp: "Formatted at 2026-09-28 03:02:59 on dist-test-slave-k5rg"
server_key: "fc70effd4bc5b9fffe98cfa8043ff4e5"
server_key_iv: "257a971638968cdd5ebf27a4078988a4"
server_key_version: "encryptionkey@0"
I20260928 03:02:59.582448 30156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:02:59.595579 30156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:02:59.599614 30156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:02:59.599865 30156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:02:59.617012 30156 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:45675
I20260928 03:02:59.617031 30222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:45675 every 8 connection(s)
I20260928 03:02:59.618350 30156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:02:59.621845 30223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:02:59.627743 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 30156
I20260928 03:02:59.627923 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:02:59.628649 13017 external_mini_cluster.cc:1428] Setting key d65ac5d761ef93d5d4b2e5822e15decf
I20260928 03:02:59.632169 30223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931: Bootstrap starting.
I20260928 03:02:59.635824 30223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931: Neither blocks nor log segments found. Creating new log.
I20260928 03:02:59.637094 30223 log.cc:824] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931: Log is configured to *not* fsync() on all Append() calls
I20260928 03:02:59.639693 30223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931: No bootstrap required, opened a new log
Sep 28 03:02:59 dist-test-slave-k5rg krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:02:59.647066 30223 raft_consensus.cc:348] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70dba8053f94a17892284f07d33d931" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45675 } }
I20260928 03:02:59.647409 30223 raft_consensus.cc:374] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:02:59.647507 30223 raft_consensus.cc:729] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d70dba8053f94a17892284f07d33d931, State: Initialized, Role: FOLLOWER
I20260928 03:02:59.648099 30223 consensus_queue.cc:260] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70dba8053f94a17892284f07d33d931" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45675 } }
I20260928 03:02:59.648386 30223 raft_consensus.cc:388] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:02:59.648533 30223 raft_consensus.cc:482] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:02:59.648711 30223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:02:59.651305 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:02:59.630552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.12.182.126:45675)
0928 03:02:59.631315 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0928 03:02:59.631329 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:02:59.631376 (+ 47us) server_negotiation.cc:407] Connection header received
0928 03:02:59.633707 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:02:59.633736 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:02:59.634262 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:02:59.634691 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:02:59.635929 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:59.637875 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:59.639179 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:02:59.639820 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:02:59.639986 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:02:59.643710 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:02:59.643718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:02:59.643731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:02:59.643768 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:02:59.646869 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:59.647541 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:59.647550 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:59.647558 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:59.647761 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:02:59.648144 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:02:59.648149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:02:59.648152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:02:59.648888 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:02:59.649159 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 03:02:59.649540 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 03:02:59.649997 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":303,"threads_started":1}
I20260928 03:02:59.651945 30223 raft_consensus.cc:504] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70dba8053f94a17892284f07d33d931" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45675 } }
I20260928 03:02:59.652484 30223 leader_election.cc:302] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d70dba8053f94a17892284f07d33d931; no voters:
I20260928 03:02:59.653195 30223 leader_election.cc:288] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:02:59.653330 30228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:02:59.654452 30228 raft_consensus.cc:686] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [term 1 LEADER]: Becoming Leader. State: Replica: d70dba8053f94a17892284f07d33d931, State: Running, Role: LEADER
I20260928 03:02:59.654968 30228 consensus_queue.cc:237] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70dba8053f94a17892284f07d33d931" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45675 } }
I20260928 03:02:59.655306 30223 sys_catalog.cc:564] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:02:59.657251 30230 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d70dba8053f94a17892284f07d33d931. Latest consensus state: current_term: 1 leader_uuid: "d70dba8053f94a17892284f07d33d931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70dba8053f94a17892284f07d33d931" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45675 } } }
I20260928 03:02:59.657616 30230 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [sys.catalog]: This master's current role is: LEADER
I20260928 03:02:59.658557 30229 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d70dba8053f94a17892284f07d33d931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70dba8053f94a17892284f07d33d931" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 45675 } } }
I20260928 03:02:59.659075 30229 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931 [sys.catalog]: This master's current role is: LEADER
I20260928 03:02:59.660727 30235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:02:59.665710 30235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:02:59.675041 30235 catalog_manager.cc:1347] Generated new cluster ID: 8cf6d048de3a4ee59d61daafc289c3e0
I20260928 03:02:59.675149 30235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:02:59.691185 30235 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:02:59.692441 30235 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:02:59.705858 30235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931: Generated new TSK 0
I20260928 03:02:59.706676 30235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 03:02:59.770888 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45675
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:00.091015 30250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:00.091300 30250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:00.091442 30250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:00.098574 30250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:00.098687 30250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:00.098840 30250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:03:00.108595 30250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:00.109803 30250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:00.111032 30250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:00.123416 30255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:00.123473 30256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:00.124033 30258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:00.124332 30250 server_base.cc:1034] running on GCE node
I20260928 03:03:00.124879 30250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:00.125808 30250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:00.127007 30250 hybrid_clock.cc:648] HybridClock initialized: now 1790564580126960 us; error 38 us; skew 500 ppm
I20260928 03:03:00.127285 30250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:00 dist-test-slave-k5rg krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564580, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:00.130244 30250 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:03:00.131578 30250 webserver.cc:460] Webserver started at http://127.12.182.65:39435/ using document root <none> and password file <none>
I20260928 03:03:00.132198 30250 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:00.132309 30250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:00.132579 30250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:00.135350 30250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "e089e7172e7a48ee9441edea3932ba6d"
format_stamp: "Formatted at 2026-09-28 03:03:00 on dist-test-slave-k5rg"
server_key: "8274b34f96567662e10703a2ab3e0b3d"
server_key_iv: "03424c79f869dc6be6f70ccb1043cb7e"
server_key_version: "encryptionkey@0"
I20260928 03:03:00.136055 30250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "e089e7172e7a48ee9441edea3932ba6d"
format_stamp: "Formatted at 2026-09-28 03:03:00 on dist-test-slave-k5rg"
server_key: "8274b34f96567662e10703a2ab3e0b3d"
server_key_iv: "03424c79f869dc6be6f70ccb1043cb7e"
server_key_version: "encryptionkey@0"
I20260928 03:03:00.140996 30250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 03:03:00.144352 30265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:00.145579 30250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:03:00.145776 30250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "e089e7172e7a48ee9441edea3932ba6d"
format_stamp: "Formatted at 2026-09-28 03:03:00 on dist-test-slave-k5rg"
server_key: "8274b34f96567662e10703a2ab3e0b3d"
server_key_iv: "03424c79f869dc6be6f70ccb1043cb7e"
server_key_version: "encryptionkey@0"
I20260928 03:03:00.145949 30250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:00.162031 30250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:00.166038 30250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:00.166307 30250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:00.167332 30250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:00.168583 30250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:00.168656 30250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:00.168776 30250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:00.168829 30250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:00.198518 30250 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:34251
I20260928 03:03:00.198572 30377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:34251 every 8 connection(s)
I20260928 03:03:00.199729 30250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:03:00.205343 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 30250
I20260928 03:03:00.205538 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 03:03:00.206287 13017 external_mini_cluster.cc:1428] Setting key a85e9965bc7c5c48cb2d298881142117
Sep 28 03:03:00 dist-test-slave-k5rg krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564580, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:00.220047 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:00.202082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:41615 (local address 127.12.182.126:45675)
0928 03:03:00.202505 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:00.202509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:00.203438 (+ 929us) server_negotiation.cc:407] Connection header received
0928 03:03:00.205244 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:00.205251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:00.205334 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:00.205427 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:00.206954 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:00.207951 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:00.210099 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:00.210495 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:00.210586 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:00.214464 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:00.214469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:00.214472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:00.214508 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:00.217077 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:00.217785 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:00.217790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:00.217793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:00.217934 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:00.218376 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:00.218380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:00.218383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:00.218689 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:00.218822 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:00.219494 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 03:03:00.219718 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":215,"threads_started":1}
I20260928 03:03:00.220876 30381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:00.202602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45675 (local address 127.12.182.65:41615)
0928 03:03:00.203283 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:00.203318 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:00.204945 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:00.205731 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:00.205740 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:00.206223 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:00.206794 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:00.206812 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:00.208096 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:00.208100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:00.209953 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:00.209969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:00.210696 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:00.210702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:00.210971 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:00.211630 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:00.211657 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:00.214237 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:00.217218 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:00.217224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:00.217237 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:00.217674 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:00.218069 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:00.218073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:00.218075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:00.218272 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:00.218815 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:00.218822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:00.219141 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 03:03:00.219361 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 03:03:00.219798 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":264,"threads_started":1}
I20260928 03:03:00.222371 30378 heartbeater.cc:347] Connected to a master server at 127.12.182.126:45675
I20260928 03:03:00.222651 30378 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:00.223356 30378 heartbeater.cc:511] Master 127.12.182.126:45675 requested a full tablet report, sending...
I20260928 03:03:00.225322 30188 ts_manager.cc:194] Registered new tserver with Master: e089e7172e7a48ee9441edea3932ba6d (127.12.182.65:34251)
I20260928 03:03:00.226822 30188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:41615
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 03:03:00.256944 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45675
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:00.569644 30384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:00.569937 30384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:00.570032 30384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:00.577203 30384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:00.577310 30384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:00.577421 30384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:03:00.587260 30384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:00.588457 30384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:00.590022 30384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:00.602439 30392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:00.602469 30390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:00.602499 30389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:00.603477 30384 server_base.cc:1034] running on GCE node
I20260928 03:03:00.604133 30384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:00.604979 30384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:00.606166 30384 hybrid_clock.cc:648] HybridClock initialized: now 1790564580606115 us; error 42 us; skew 500 ppm
I20260928 03:03:00.606459 30384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:00 dist-test-slave-k5rg krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564580, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:00.609359 30384 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:03:00.610787 30384 webserver.cc:460] Webserver started at http://127.12.182.66:34069/ using document root <none> and password file <none>
I20260928 03:03:00.611426 30384 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:00.611538 30384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:00.611824 30384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:00.614660 30384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "c1b32ee990cc44ddbc10d5d20e5f18be"
format_stamp: "Formatted at 2026-09-28 03:03:00 on dist-test-slave-k5rg"
server_key: "fcc4cf174d538209bd205f9325d63e88"
server_key_iv: "f29c015bb3f7e4d4841214b765be8ec3"
server_key_version: "encryptionkey@0"
I20260928 03:03:00.615396 30384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "c1b32ee990cc44ddbc10d5d20e5f18be"
format_stamp: "Formatted at 2026-09-28 03:03:00 on dist-test-slave-k5rg"
server_key: "fcc4cf174d538209bd205f9325d63e88"
server_key_iv: "f29c015bb3f7e4d4841214b765be8ec3"
server_key_version: "encryptionkey@0"
I20260928 03:03:00.620518 30384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 03:03:00.623930 30399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:00.625243 30384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 03:03:00.625388 30384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "c1b32ee990cc44ddbc10d5d20e5f18be"
format_stamp: "Formatted at 2026-09-28 03:03:00 on dist-test-slave-k5rg"
server_key: "fcc4cf174d538209bd205f9325d63e88"
server_key_iv: "f29c015bb3f7e4d4841214b765be8ec3"
server_key_version: "encryptionkey@0"
I20260928 03:03:00.625564 30384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:00.639259 30384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:00.643116 30384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:00.643381 30384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:00.644279 30384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:00.645506 30384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:00.645583 30384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:00.645733 30384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:00.645803 30384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:00.677201 30384 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:35959
I20260928 03:03:00.677237 30511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:35959 every 8 connection(s)
I20260928 03:03:00.678468 30384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:03:00.687312 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 30384
I20260928 03:03:00.687467 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 03:03:00.687868 13017 external_mini_cluster.cc:1428] Setting key d6eee53d6779a823970a75b90ffc14a2
Sep 28 03:03:00 dist-test-slave-k5rg krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564580, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:00.697158 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:00.680914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:57845 (local address 127.12.182.126:45675)
0928 03:03:00.681094 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:00.681098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:00.682162 (+ 1064us) server_negotiation.cc:407] Connection header received
0928 03:03:00.683955 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:00.683961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:00.684043 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:00.684133 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:00.685596 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:00.686398 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:00.688068 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:00.688550 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:00.688665 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:00.691932 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:00.691937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:00.691940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:00.691971 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:00.694334 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:00.694987 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:00.694992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:00.694994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:00.695113 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:00.695506 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:00.695511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:00.695514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:00.695770 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:00.695924 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:00.696597 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 03:03:00.696826 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 03:03:00.698140 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:00.681305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45675 (local address 127.12.182.66:57845)
0928 03:03:00.682011 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:00.682047 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:00.683692 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:00.684376 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:00.684384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:00.684886 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:00.685431 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:00.685448 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:00.686546 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:00.686550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:00.687934 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:00.687949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:00.688773 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:00.688781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:00.688993 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:00.689704 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:00.689726 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:00.691726 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:00.694462 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:00.694471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:00.694486 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:00.694881 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:00.695211 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:00.695216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:00.695220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:00.695403 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:00.695900 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:00.695908 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:00.696230 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 03:03:00.696491 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 03:03:00.697005 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":291,"threads_started":1}
I20260928 03:03:00.699438 30512 heartbeater.cc:347] Connected to a master server at 127.12.182.126:45675
I20260928 03:03:00.699747 30512 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:00.700541 30512 heartbeater.cc:511] Master 127.12.182.126:45675 requested a full tablet report, sending...
I20260928 03:03:00.701974 30188 ts_manager.cc:194] Registered new tserver with Master: c1b32ee990cc44ddbc10d5d20e5f18be (127.12.182.66:35959)
I20260928 03:03:00.702749 30188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:57845
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 03:03:00.736778 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45675
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:01.059263 30517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:01.059661 30517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:01.059821 30517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:01.069378 30517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:01.069514 30517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:01.069736 30517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:03:01.083900 30517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:45675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:01.085635 30517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:01.087404 30517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:01.102123 30522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:01.102154 30523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:01.102665 30525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:01.102895 30517 server_base.cc:1034] running on GCE node
I20260928 03:03:01.103440 30517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:01.104300 30517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:01.105499 30517 hybrid_clock.cc:648] HybridClock initialized: now 1790564581105441 us; error 48 us; skew 500 ppm
I20260928 03:03:01.105834 30517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:01 dist-test-slave-k5rg krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564581, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:01.108665 30517 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:03:01.110028 30517 webserver.cc:460] Webserver started at http://127.12.182.67:39711/ using document root <none> and password file <none>
I20260928 03:03:01.110627 30517 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:01.110733 30517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:01.111003 30517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:01.113888 30517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "973c873f12c0460dbe991edf2a4e6345"
format_stamp: "Formatted at 2026-09-28 03:03:01 on dist-test-slave-k5rg"
server_key: "1e09858efcd963d348faeb5e8631449f"
server_key_iv: "3bb944c0401fa95df5a46169b5749864"
server_key_version: "encryptionkey@0"
I20260928 03:03:01.114573 30517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "973c873f12c0460dbe991edf2a4e6345"
format_stamp: "Formatted at 2026-09-28 03:03:01 on dist-test-slave-k5rg"
server_key: "1e09858efcd963d348faeb5e8631449f"
server_key_iv: "3bb944c0401fa95df5a46169b5749864"
server_key_version: "encryptionkey@0"
I20260928 03:03:01.119424 30517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:03:01.122691 30532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:01.123943 30517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:03:01.124079 30517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "973c873f12c0460dbe991edf2a4e6345"
format_stamp: "Formatted at 2026-09-28 03:03:01 on dist-test-slave-k5rg"
server_key: "1e09858efcd963d348faeb5e8631449f"
server_key_iv: "3bb944c0401fa95df5a46169b5749864"
server_key_version: "encryptionkey@0"
I20260928 03:03:01.124250 30517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:01.142448 30517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:01.146101 30517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:01.146353 30517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:01.147264 30517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:01.148541 30517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:01.148609 30517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:01.148725 30517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:01.148772 30517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:01.177923 30517 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:41261
I20260928 03:03:01.177986 30644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:41261 every 8 connection(s)
I20260928 03:03:01.179096 30517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:03:01.187753 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 30517
I20260928 03:03:01.187940 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 03:03:01.188479 13017 external_mini_cluster.cc:1428] Setting key 3423afa4d6f349f962d0c174ac1b6eb5
Sep 28 03:03:01 dist-test-slave-k5rg krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564581, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:01.198175 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.181338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:42305 (local address 127.12.182.126:45675)
0928 03:03:01.181533 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.181539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.182714 (+ 1175us) server_negotiation.cc:407] Connection header received
0928 03:03:01.184419 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.184424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.184489 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.184570 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:01.186024 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.186778 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.188727 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.189207 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.189304 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.192780 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:01.192786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:01.192789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:01.192821 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:01.195307 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:01.195971 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:01.195976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:01.195978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:01.196093 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:01.196490 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:01.196494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:01.196496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:01.196759 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:01.196887 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.197538 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.197804 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:03:01.199013 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.181850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:45675 (local address 127.12.182.67:42305)
0928 03:03:01.182556 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.182591 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:01.184196 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:01.184743 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:01.184750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:01.185235 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:01.185882 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.185902 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.186907 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.186912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.188574 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.188589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.189414 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.189422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.189664 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.190357 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:01.190393 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:01.192557 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:01.195445 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:01.195453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:01.195469 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:01.195852 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:01.196199 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:01.196204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:01.196208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:01.196380 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:01.196877 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:01.196884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:01.197195 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 03:03:01.197421 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 03:03:01.197926 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":281,"threads_started":1}
I20260928 03:03:01.200376 30645 heartbeater.cc:347] Connected to a master server at 127.12.182.126:45675
I20260928 03:03:01.200677 30645 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:01.201457 30645 heartbeater.cc:511] Master 127.12.182.126:45675 requested a full tablet report, sending...
I20260928 03:03:01.203032 30188 ts_manager.cc:194] Registered new tserver with Master: 973c873f12c0460dbe991edf2a4e6345 (127.12.182.67:41261)
I20260928 03:03:01.203840 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:03:01.204051 30188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:42305
I20260928 03:03:01.219849 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.208201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.12.182.126:45675)
0928 03:03:01.208330 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.208334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.208749 (+ 415us) server_negotiation.cc:407] Connection header received
0928 03:03:01.208868 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.208872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.208926 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.209000 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:01.210312 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.211021 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.212466 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.212899 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.212989 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.215129 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:01.215135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:01.215138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:01.215171 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:01.217525 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:01.218187 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:01.218192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:01.218194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:01.218324 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:01.218655 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:01.218659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:01.218661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:01.218988 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:01.219168 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.219454 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.219605 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 03:03:01.223570 30188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 03:03:01.229490 30378 heartbeater.cc:503] Master 127.12.182.126:45675 was elected leader, sending a full tablet report...
W20260928 03:03:01.230613 30188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:03:01.254482 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.244335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.12.182.67:41261)
0928 03:03:01.245994 (+ 1659us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.246003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.246038 (+ 35us) server_negotiation.cc:407] Connection header received
0928 03:03:01.246143 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.246152 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.246516 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.246754 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:01.247736 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.249307 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.251791 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.253034 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.253152 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.253350 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.253987 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.254155 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1476,"thread_start_us":1319,"threads_started":1}
I20260928 03:03:01.254901 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.244889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.12.182.66:35959)
0928 03:03:01.246614 (+ 1725us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.246622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.247157 (+ 535us) server_negotiation.cc:407] Connection header received
0928 03:03:01.247489 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.247497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.247740 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.247889 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:01.248632 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.250302 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.252570 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.253785 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.253903 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.254077 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.254285 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.254450 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1580,"thread_start_us":1445,"threads_started":1}
I20260928 03:03:01.255074 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.244355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:35959 (local address 127.0.0.1:55414)
0928 03:03:01.247038 (+ 2683us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.247056 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:01.247279 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:01.248107 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:01.248110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:01.248128 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:01.248444 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.248456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.250461 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.250466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.252398 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.252413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.253971 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.253978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.254128 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.254157 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 03:03:01.254234 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 03:03:01.254430 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2531,"mutex_wait_us":3,"thread_start_us":369,"threads_started":1}
I20260928 03:03:01.255836 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.247125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.12.182.65:34251)
0928 03:03:01.248072 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.248080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.248105 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:03:01.248197 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.248203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.248430 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.249236 (+ 806us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:01.249489 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.251008 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.253358 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.254729 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.255042 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.255272 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.255448 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.255607 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":632,"threads_started":1}
I20260928 03:03:01.255864 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.244104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:41261 (local address 127.0.0.1:38198)
0928 03:03:01.245073 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.245126 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:01.245325 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:01.247030 (+ 1705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:01.247041 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:01.247079 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:01.247539 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.247550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.249484 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.249488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.251644 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.251656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.253478 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.253485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.253716 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.253771 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 03:03:01.254781 (+ 1010us) client_negotiation.cc:240] Negotiation successful
0928 03:03:01.254929 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":27,"thread_start_us":2649,"threads_started":1}
I20260928 03:03:01.256219 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.246492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:34251 (local address 127.0.0.1:36068)
0928 03:03:01.247378 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.247398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:01.247510 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:01.248621 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:01.248626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:01.248651 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:01.249060 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.249070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.251150 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.251156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.253197 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.253212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.254964 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.254970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.255237 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.255269 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 03:03:01.255353 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 03:03:01.255476 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":848,"threads_started":1}
I20260928 03:03:01.259258 30580 tablet_service.cc:1467] Processing CreateTablet for tablet f15ef303d6224bd78c976425dd4a956c (DEFAULT_TABLE table=test-table [id=935f344114b346868e1f793ee765b027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:01.259282 30447 tablet_service.cc:1467] Processing CreateTablet for tablet f15ef303d6224bd78c976425dd4a956c (DEFAULT_TABLE table=test-table [id=935f344114b346868e1f793ee765b027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:01.260540 30580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15ef303d6224bd78c976425dd4a956c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:01.260610 30447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15ef303d6224bd78c976425dd4a956c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:01.260303 30313 tablet_service.cc:1467] Processing CreateTablet for tablet f15ef303d6224bd78c976425dd4a956c (DEFAULT_TABLE table=test-table [id=935f344114b346868e1f793ee765b027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:01.261583 30313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15ef303d6224bd78c976425dd4a956c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:01.271593 30663 tablet_bootstrap.cc:492] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: Bootstrap starting.
I20260928 03:03:01.274761 30663 tablet_bootstrap.cc:654] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:01.274921 30664 tablet_bootstrap.cc:492] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: Bootstrap starting.
I20260928 03:03:01.277231 30663 log.cc:824] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:01.278235 30665 tablet_bootstrap.cc:492] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d: Bootstrap starting.
I20260928 03:03:01.278564 30664 tablet_bootstrap.cc:654] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:01.279801 30664 log.cc:824] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:01.281349 30665 tablet_bootstrap.cc:654] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:01.282354 30663 tablet_bootstrap.cc:492] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: No bootstrap required, opened a new log
I20260928 03:03:01.282593 30663 ts_tablet_manager.cc:1397] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 03:03:01.282589 30665 log.cc:824] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:01.282692 30664 tablet_bootstrap.cc:492] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: No bootstrap required, opened a new log
I20260928 03:03:01.282953 30664 ts_tablet_manager.cc:1397] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260928 03:03:01.285436 30665 tablet_bootstrap.cc:492] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d: No bootstrap required, opened a new log
I20260928 03:03:01.285733 30665 ts_tablet_manager.cc:1397] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260928 03:03:01.290483 30664 raft_consensus.cc:348] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 } } peers { permanent_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35959 } } peers { permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 } }
I20260928 03:03:01.290838 30664 raft_consensus.cc:374] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:01.290926 30664 raft_consensus.cc:729] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b32ee990cc44ddbc10d5d20e5f18be, State: Initialized, Role: FOLLOWER
I20260928 03:03:01.291496 30664 consensus_queue.cc:260] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 } } peers { permanent_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35959 } } peers { permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 } }
I20260928 03:03:01.292551 30512 heartbeater.cc:503] Master 127.12.182.126:45675 was elected leader, sending a full tablet report...
I20260928 03:03:01.293069 30663 raft_consensus.cc:348] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 } } peers { permanent_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35959 } } peers { permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 } }
I20260928 03:03:01.293557 30663 raft_consensus.cc:374] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:01.293696 30663 raft_consensus.cc:729] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973c873f12c0460dbe991edf2a4e6345, State: Initialized, Role: FOLLOWER
I20260928 03:03:01.294350 30663 consensus_queue.cc:260] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 } } peers { permanent_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35959 } } peers { permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 } }
I20260928 03:03:01.294497 30664 ts_tablet_manager.cc:1428] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260928 03:03:01.296235 30665 raft_consensus.cc:348] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 } } peers { permanent_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35959 } } peers { permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 } }
I20260928 03:03:01.296702 30665 raft_consensus.cc:374] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:01.296789 30665 raft_consensus.cc:729] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e089e7172e7a48ee9441edea3932ba6d, State: Initialized, Role: FOLLOWER
I20260928 03:03:01.297451 30665 consensus_queue.cc:260] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 } } peers { permanent_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35959 } } peers { permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 } }
I20260928 03:03:01.298345 30645 heartbeater.cc:503] Master 127.12.182.126:45675 was elected leader, sending a full tablet report...
I20260928 03:03:01.299106 30663 ts_tablet_manager.cc:1428] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260928 03:03:01.303133 30665 ts_tablet_manager.cc:1428] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
W20260928 03:03:01.430591 30646 tablet.cc:2369] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:03:01.430591 30513 tablet.cc:2369] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:03:01.451890 30379 tablet.cc:2369] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:03:01.462509 30669 raft_consensus.cc:482] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:03:01.462808 30669 raft_consensus.cc:504] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 } } peers { permanent_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35959 } } peers { permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 } }
I20260928 03:03:01.464490 30669 leader_election.cc:288] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 973c873f12c0460dbe991edf2a4e6345 (127.12.182.67:41261), e089e7172e7a48ee9441edea3932ba6d (127.12.182.65:34251)
I20260928 03:03:01.473170 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.464911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:38651 (local address 127.12.182.65:34251)
0928 03:03:01.465120 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.465126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.465802 (+ 676us) server_negotiation.cc:407] Connection header received
0928 03:03:01.466026 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.466030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.466111 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.466204 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:01.466894 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.467980 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.470928 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.472151 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.472317 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.472393 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.472781 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.472913 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:03:01.473167 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.464978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:52823 (local address 127.12.182.67:41261)
0928 03:03:01.465144 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.465149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.465855 (+ 706us) server_negotiation.cc:407] Connection header received
0928 03:03:01.466025 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.466030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.466111 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.466220 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:01.466894 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.467980 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.470920 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.472171 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.472323 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.472393 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.472780 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.472913 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:03:01.473397 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.464773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:34251 (local address 127.12.182.66:38651)
0928 03:03:01.465686 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.465706 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:01.465892 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:01.466234 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:01.466239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:01.466274 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:01.466742 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.466754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.468100 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.468106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.470749 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.470766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.472412 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.472418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.472607 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.472651 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 03:03:01.472744 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 03:03:01.472928 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":143,"spinlock_wait_cycles":26880,"thread_start_us":589,"threads_started":1}
I20260928 03:03:01.473742 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.464846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:41261 (local address 127.12.182.66:52823)
0928 03:03:01.465753 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.465766 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:01.465892 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:01.466234 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:01.466239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:01.466271 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:01.466752 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.466761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.468102 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.468106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.470755 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:01.470766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.472420 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.472426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:01.472600 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.472659 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 03:03:01.472744 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 03:03:01.472928 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":227,"spinlock_wait_cycles":16640,"thread_start_us":488,"threads_started":1}
I20260928 03:03:01.474402 30333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15ef303d6224bd78c976425dd4a956c" candidate_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e089e7172e7a48ee9441edea3932ba6d" is_pre_election: true
I20260928 03:03:01.474699 30600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15ef303d6224bd78c976425dd4a956c" candidate_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973c873f12c0460dbe991edf2a4e6345" is_pre_election: true
I20260928 03:03:01.474885 30333 raft_consensus.cc:2454] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b32ee990cc44ddbc10d5d20e5f18be in term 0.
I20260928 03:03:01.475147 30600 raft_consensus.cc:2454] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b32ee990cc44ddbc10d5d20e5f18be in term 0.
I20260928 03:03:01.475513 30403 leader_election.cc:302] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1b32ee990cc44ddbc10d5d20e5f18be, e089e7172e7a48ee9441edea3932ba6d; no voters:
I20260928 03:03:01.475884 30669 raft_consensus.cc:2781] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:03:01.476042 30669 raft_consensus.cc:482] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:03:01.476147 30669 raft_consensus.cc:3037] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:01.478303 30669 raft_consensus.cc:504] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 } } peers { permanent_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35959 } } peers { permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 } }
I20260928 03:03:01.478873 30669 leader_election.cc:288] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [CANDIDATE]: Term 1 election: Requested vote from peers 973c873f12c0460dbe991edf2a4e6345 (127.12.182.67:41261), e089e7172e7a48ee9441edea3932ba6d (127.12.182.65:34251)
I20260928 03:03:01.479334 30600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15ef303d6224bd78c976425dd4a956c" candidate_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973c873f12c0460dbe991edf2a4e6345"
I20260928 03:03:01.479337 30333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15ef303d6224bd78c976425dd4a956c" candidate_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e089e7172e7a48ee9441edea3932ba6d"
I20260928 03:03:01.479514 30333 raft_consensus.cc:3037] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:01.479504 30600 raft_consensus.cc:3037] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:01.481904 30600 raft_consensus.cc:2454] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b32ee990cc44ddbc10d5d20e5f18be in term 1.
I20260928 03:03:01.481946 30333 raft_consensus.cc:2454] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b32ee990cc44ddbc10d5d20e5f18be in term 1.
I20260928 03:03:01.482323 30403 leader_election.cc:302] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1b32ee990cc44ddbc10d5d20e5f18be, e089e7172e7a48ee9441edea3932ba6d; no voters:
I20260928 03:03:01.482599 30669 raft_consensus.cc:2781] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:01.483039 30669 raft_consensus.cc:686] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 1 LEADER]: Becoming Leader. State: Replica: c1b32ee990cc44ddbc10d5d20e5f18be, State: Running, Role: LEADER
I20260928 03:03:01.483579 30669 consensus_queue.cc:237] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 } } peers { permanent_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35959 } } peers { permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 } }
I20260928 03:03:01.488032 30187 catalog_manager.cc:5495] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be reported cstate change: term changed from 0 to 1, leader changed from <none> to c1b32ee990cc44ddbc10d5d20e5f18be (127.12.182.66). New cstate: current_term: 1 leader_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b32ee990cc44ddbc10d5d20e5f18be" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 } health_report { overall_health: UNKNOWN } } }
I20260928 03:03:01.551750 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.546215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.12.182.66:35959)
0928 03:03:01.546431 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.546437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.546456 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:03:01.546694 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.546698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.546796 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.546929 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:01.547304 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.548039 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.549451 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.549908 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.550176 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.550655 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:01.551175 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:01.551303 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.551407 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.551487 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:03:01.559646 30600 raft_consensus.cc:1261] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345 [term 1 FOLLOWER]: Refusing update from remote peer c1b32ee990cc44ddbc10d5d20e5f18be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:01.559655 30333 raft_consensus.cc:1261] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d [term 1 FOLLOWER]: Refusing update from remote peer c1b32ee990cc44ddbc10d5d20e5f18be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:01.560665 30674 consensus_queue.cc:1035] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [LEADER]: Connected to new peer: Peer: permanent_uuid: "973c873f12c0460dbe991edf2a4e6345" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 41261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:01.561058 30669 consensus_queue.cc:1035] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e089e7172e7a48ee9441edea3932ba6d" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:01.571480 30677 mvcc.cc:205] Tried to move back new op lower bound from 7334152526058299392 to 7334152525760225280. Current Snapshot: MvccSnapshot[applied={T|T < 7334152526058299392}]
I20260928 03:03:01.573865 30679 mvcc.cc:205] Tried to move back new op lower bound from 7334152526058299392 to 7334152525760225280. Current Snapshot: MvccSnapshot[applied={T|T < 7334152526058299392}]
I20260928 03:03:01.584575 30188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:01.584818 30188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:01.588223 30188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d70dba8053f94a17892284f07d33d931: Sending DeleteTablet for 3 replicas of tablet f15ef303d6224bd78c976425dd4a956c
I20260928 03:03:01.589345 30580 tablet_service.cc:1514] Processing DeleteTablet for tablet f15ef303d6224bd78c976425dd4a956c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:01 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:38198
I20260928 03:03:01.589479 30447 tablet_service.cc:1514] Processing DeleteTablet for tablet f15ef303d6224bd78c976425dd4a956c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:01 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:55414
I20260928 03:03:01.590184 30686 tablet_replica.cc:330] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: stopping tablet replica
I20260928 03:03:01.590420 30687 tablet_replica.cc:330] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: stopping tablet replica
I20260928 03:03:01.590783 30687 raft_consensus.cc:2229] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:03:01.590819 30686 raft_consensus.cc:2229] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:01.591369 30686 raft_consensus.cc:2258] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:01.591409 30687 raft_consensus.cc:2258] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:01.591523 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 30250
I20260928 03:03:01.593684 30686 ts_tablet_manager.cc:1905] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:01.593683 30687 ts_tablet_manager.cc:1905] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:01.594692 30313 tablet_service.cc:1514] Processing DeleteTablet for tablet f15ef303d6224bd78c976425dd4a956c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:01 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:36068
I20260928 03:03:01.600407 30686 ts_tablet_manager.cc:1918] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:01.600628 30687 ts_tablet_manager.cc:1918] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:01.600787 30687 log.cc:1196] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/f15ef303d6224bd78c976425dd4a956c
I20260928 03:03:01.601361 30687 ts_tablet_manager.cc:1939] T f15ef303d6224bd78c976425dd4a956c P c1b32ee990cc44ddbc10d5d20e5f18be: Deleting consensus metadata
I20260928 03:03:01.601384 30686 log.cc:1196] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/f15ef303d6224bd78c976425dd4a956c
I20260928 03:03:01.602007 30686 ts_tablet_manager.cc:1939] T f15ef303d6224bd78c976425dd4a956c P 973c873f12c0460dbe991edf2a4e6345: Deleting consensus metadata
I20260928 03:03:01.603164 30173 catalog_manager.cc:4841] TS c1b32ee990cc44ddbc10d5d20e5f18be (127.12.182.66:35959): tablet f15ef303d6224bd78c976425dd4a956c (table test-table [id=935f344114b346868e1f793ee765b027]) successfully deleted
I20260928 03:03:01.604228 30174 catalog_manager.cc:4841] TS 973c873f12c0460dbe991edf2a4e6345 (127.12.182.67:41261): tablet f15ef303d6224bd78c976425dd4a956c (table test-table [id=935f344114b346868e1f793ee765b027]) successfully deleted
Sep 28 03:03:01 dist-test-slave-k5rg krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 03:03:01.609550 30689 tablet_replica.cc:330] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d: stopping tablet replica
I20260928 03:03:01.610296 30689 raft_consensus.cc:2229] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:01.610881 30689 raft_consensus.cc:2258] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:01.613083 30689 ts_tablet_manager.cc:1905] T f15ef303d6224bd78c976425dd4a956c P e089e7172e7a48ee9441edea3932ba6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:01.616228 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.593105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.12.182.65:34251)
0928 03:03:01.593286 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.593292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.601114 (+ 7822us) server_negotiation.cc:407] Connection header received
0928 03:03:01.601308 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.601314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.601400 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.601503 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:01.603246 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.604415 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.605887 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.606327 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.606399 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.610563 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:01.610576 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:01.610590 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:01.610628 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:01.613524 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:01.614239 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:01.614246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:01.614249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:01.614390 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:01.614743 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:01.614747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:01.614749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:01.615111 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:01.615399 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.615689 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.615911 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 03:03:01.616873 30373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:01.708628 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 30250
I20260928 03:03:01.723001 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 30384
W20260928 03:03:01.722930 30175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.182.65:34251 (error 108)
I20260928 03:03:01.724642 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.724212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:34251 (local address 127.0.0.1:36096)
0928 03:03:01.724381 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.724503 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 03:03:01.724871 30175 catalog_manager.cc:4584] TS e089e7172e7a48ee9441edea3932ba6d (127.12.182.65:34251): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f15ef303d6224bd78c976425dd4a956c: Network error: Client connection negotiation failed: client connection to 127.12.182.65:34251: connect: Connection refused (error 111)
Sep 28 03:03:01 dist-test-slave-k5rg krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 03:03:01.739763 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.723927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.12.182.66:35959)
0928 03:03:01.724111 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.724117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.724146 (+ 29us) server_negotiation.cc:407] Connection header received
0928 03:03:01.724209 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.724214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.724290 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.724388 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:01.725986 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.726852 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.728688 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.729185 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.729275 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.734172 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:01.734190 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:01.734204 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:01.734241 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:01.737023 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:01.737745 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:01.737755 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:01.737761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:01.737899 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:01.738259 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:01.738265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:01.738268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:01.738627 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:01.738946 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.739199 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.739416 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:03:01.740423 30507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:01.778363 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.777835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:34251 (local address 127.0.0.1:36104)
0928 03:03:01.778071 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.778186 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 03:03:01.779425 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.779037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:34251 (local address 127.0.0.1:36120)
0928 03:03:01.779188 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.779276 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 03:03:01.840288 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.839641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:34251 (local address 127.0.0.1:36132)
0928 03:03:01.839940 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.840055 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260928 03:03:01.841326 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.840955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:34251 (local address 127.0.0.1:36140)
0928 03:03:01.841097 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.841182 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 03:03:01.850677 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 30384
I20260928 03:03:01.865413 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 30517
Sep 28 03:03:01 dist-test-slave-k5rg krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 03:03:01.879078 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.866176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.12.182.67:41261)
0928 03:03:01.866362 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:01.866366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:01.866385 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:03:01.866440 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:01.866444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:01.866506 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:01.866585 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:01.867827 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.868548 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.869790 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:01.870155 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:01.870244 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:01.874357 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:01.874371 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:01.874386 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:01.874425 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:01.876919 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:01.877480 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:01.877486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:01.877489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:01.877626 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:01.877923 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:01.877926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:01.877928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:01.878280 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:01.878480 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:01.878683 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 03:03:01.878835 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 03:03:01.879642 30640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:01.921875 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.921366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:34251 (local address 127.0.0.1:36144)
0928 03:03:01.921575 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.921700 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 03:03:01.922938 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:01.922577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:34251 (local address 127.0.0.1:36148)
0928 03:03:01.922756 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:01.922826 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 03:03:01.965905 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 30517
I20260928 03:03:01.979050 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 30156
I20260928 03:03:01.980224 30218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:02.089649 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 30156
2026-09-28T03:03:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4140 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T03:03:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:02Z Disabled control of system clock
I20260928 03:03:02.157393 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:46105
--webserver_interface=127.12.182.126
--webserver_port=0
--builtin_ntp_servers=127.12.182.84:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:46105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:03:02.467921 30698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:02.468211 30698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:02.468297 30698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:02.475371 30698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:02.475466 30698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:02.475508 30698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:02.475567 30698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:02.475610 30698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:02.484530 30698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:46105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:46105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:02.485705 30698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:02.487453 30698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:02.498310 30703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:02.498396 30704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:02.498368 30706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:02.499280 30698 server_base.cc:1034] running on GCE node
I20260928 03:03:02.499969 30698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:02.501209 30698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:02.502410 30698 hybrid_clock.cc:648] HybridClock initialized: now 1790564582502359 us; error 39 us; skew 500 ppm
I20260928 03:03:02.502799 30698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:02.505051 30698 webserver.cc:460] Webserver started at http://127.12.182.126:39125/ using document root <none> and password file <none>
I20260928 03:03:02.505734 30698 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:02.505831 30698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:02.506119 30698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:02.508872 30698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "03a5ccc577984ce5a5fafae44a41a6a7"
format_stamp: "Formatted at 2026-09-28 03:03:02 on dist-test-slave-k5rg"
server_key: "7b357a4a8f9e5eedabc777bd74cd0c2b"
server_key_iv: "f72a7effe744bb6cec6b5b7b9b6d42dd"
server_key_version: "encryptionkey@0"
I20260928 03:03:02.509639 30698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "03a5ccc577984ce5a5fafae44a41a6a7"
format_stamp: "Formatted at 2026-09-28 03:03:02 on dist-test-slave-k5rg"
server_key: "7b357a4a8f9e5eedabc777bd74cd0c2b"
server_key_iv: "f72a7effe744bb6cec6b5b7b9b6d42dd"
server_key_version: "encryptionkey@0"
I20260928 03:03:02.514287 30698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:03:02.517508 30712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:02.518785 30698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:03:02.518939 30698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "03a5ccc577984ce5a5fafae44a41a6a7"
format_stamp: "Formatted at 2026-09-28 03:03:02 on dist-test-slave-k5rg"
server_key: "7b357a4a8f9e5eedabc777bd74cd0c2b"
server_key_iv: "f72a7effe744bb6cec6b5b7b9b6d42dd"
server_key_version: "encryptionkey@0"
I20260928 03:03:02.519124 30698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:02.554102 30698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:02.554908 30698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:02.555167 30698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:02.572351 30698 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:46105
I20260928 03:03:02.572372 30763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:46105 every 8 connection(s)
I20260928 03:03:02.573653 30698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:02.577323 30764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:02.577562 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 30698
I20260928 03:03:02.577757 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:02.578379 13017 external_mini_cluster.cc:1428] Setting key 511f5060a5b474c781ed5d975ee72601
I20260928 03:03:02.590636 30764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7: Bootstrap starting.
I20260928 03:03:02.591799 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:02.580182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.12.182.126:46105)
0928 03:03:02.581159 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:02.581169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:02.581209 (+ 40us) server_negotiation.cc:407] Connection header received
0928 03:03:02.583334 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:02.583356 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:02.583800 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:02.584120 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:02.584679 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:02.585936 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:02.587764 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:02.588431 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:02.588555 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:02.589755 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:02.589762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:02.589792 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:02.590283 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:02.590355 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:02.590379 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:02.590460 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:02.590677 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 03:03:02.590949 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":378,"threads_started":1}
I20260928 03:03:02.593813 30764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:02.594893 30764 log.cc:824] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:02.597334 30764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7: No bootstrap required, opened a new log
I20260928 03:03:02.604518 30764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5ccc577984ce5a5fafae44a41a6a7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46105 } }
I20260928 03:03:02.604888 30764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:02.604996 30764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a5ccc577984ce5a5fafae44a41a6a7, State: Initialized, Role: FOLLOWER
I20260928 03:03:02.605571 30764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5ccc577984ce5a5fafae44a41a6a7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46105 } }
I20260928 03:03:02.605875 30764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:02.605984 30764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:02.606102 30764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:02.608075 30764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5ccc577984ce5a5fafae44a41a6a7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46105 } }
I20260928 03:03:02.608507 30764 leader_election.cc:302] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03a5ccc577984ce5a5fafae44a41a6a7; no voters:
I20260928 03:03:02.609046 30764 leader_election.cc:288] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:02.609112 30769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:02.609786 30769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [term 1 LEADER]: Becoming Leader. State: Replica: 03a5ccc577984ce5a5fafae44a41a6a7, State: Running, Role: LEADER
I20260928 03:03:02.610257 30769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5ccc577984ce5a5fafae44a41a6a7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46105 } }
I20260928 03:03:02.610993 30764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:02.612612 30770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03a5ccc577984ce5a5fafae44a41a6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5ccc577984ce5a5fafae44a41a6a7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46105 } } }
I20260928 03:03:02.613018 30770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:02.616165 30771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03a5ccc577984ce5a5fafae44a41a6a7. Latest consensus state: current_term: 1 leader_uuid: "03a5ccc577984ce5a5fafae44a41a6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5ccc577984ce5a5fafae44a41a6a7" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 46105 } } }
I20260928 03:03:02.616477 30771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:02.616822 30777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:02.622243 30777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:02.632054 30777 catalog_manager.cc:1347] Generated new cluster ID: e5de873f216947859b822294f347a7bf
I20260928 03:03:02.632171 30777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:02.651697 30777 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:02.652889 30777 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:02.661410 30777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7: Generated new TSK 0
I20260928 03:03:02.662184 30777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:02.673650 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46105
--builtin_ntp_servers=127.12.182.84:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:03:02.991962 30788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:02.992261 30788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:02.992378 30788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:02.999821 30788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:02.999925 30788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:03.000061 30788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:03:03.009243 30788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:03.010515 30788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:03.012306 30788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:03.025096 30793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:03.025063 30794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:03.025120 30796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:03.026247 30788 server_base.cc:1034] running on GCE node
I20260928 03:03:03.026829 30788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:03.027762 30788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:03.028960 30788 hybrid_clock.cc:648] HybridClock initialized: now 1790564583028910 us; error 41 us; skew 500 ppm
I20260928 03:03:03.029258 30788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:03.036921 30788 webserver.cc:460] Webserver started at http://127.12.182.65:42945/ using document root <none> and password file <none>
I20260928 03:03:03.037665 30788 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:03.037799 30788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:03.038110 30788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:03.041007 30788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "6e3ffd679398410eb94e2e479d5c420b"
format_stamp: "Formatted at 2026-09-28 03:03:03 on dist-test-slave-k5rg"
server_key: "7193f86d5de64befc1cdf4afdf4067f7"
server_key_iv: "65ae4c08982b3ac611c803759be1b78d"
server_key_version: "encryptionkey@0"
I20260928 03:03:03.041858 30788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "6e3ffd679398410eb94e2e479d5c420b"
format_stamp: "Formatted at 2026-09-28 03:03:03 on dist-test-slave-k5rg"
server_key: "7193f86d5de64befc1cdf4afdf4067f7"
server_key_iv: "65ae4c08982b3ac611c803759be1b78d"
server_key_version: "encryptionkey@0"
I20260928 03:03:03.046877 30788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 03:03:03.050272 30802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:03.051550 30788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:03:03.051721 30788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "6e3ffd679398410eb94e2e479d5c420b"
format_stamp: "Formatted at 2026-09-28 03:03:03 on dist-test-slave-k5rg"
server_key: "7193f86d5de64befc1cdf4afdf4067f7"
server_key_iv: "65ae4c08982b3ac611c803759be1b78d"
server_key_version: "encryptionkey@0"
I20260928 03:03:03.051918 30788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:03.072043 30788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:03.073154 30788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:03.073494 30788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:03.074627 30788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:03.076483 30788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:03.076572 30788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:03.076651 30788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:03.076711 30788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:03.107407 30788 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:39571
I20260928 03:03:03.107477 30914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:39571 every 8 connection(s)
I20260928 03:03:03.108704 30788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:03:03.118883 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 30788
I20260928 03:03:03.119056 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 03:03:03.119485 13017 external_mini_cluster.cc:1428] Setting key 5bb9d24777cc61c5ebe7de85f56a4ddd
I20260928 03:03:03.120596 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:03.111168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:43471 (local address 127.12.182.126:46105)
0928 03:03:03.111606 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:03.111611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:03.112649 (+ 1038us) server_negotiation.cc:407] Connection header received
0928 03:03:03.114428 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:03.114434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:03.114519 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:03.114610 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:03.115611 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.116351 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.118015 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.118418 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.118523 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:03.119801 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:03.119807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:03.119810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:03.119828 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:03.119872 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:03.119884 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:03.119954 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:03.120169 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 03:03:03.120265 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260928 03:03:03.121598 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:03.111627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46105 (local address 127.12.182.65:43471)
0928 03:03:03.112509 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:03.112543 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:03.114170 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:03.114838 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:03.114845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:03.114927 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:03.115456 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:03.115472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.116495 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.116499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:03.117892 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:03.117907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.118623 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.118639 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:03.118853 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:03.119479 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:03:03.119481 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:03.119518 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:03:03.119618 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:03.119967 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:03.119978 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:03.119987 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 03:03:03.120139 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 03:03:03.120510 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":294,"threads_started":1}
I20260928 03:03:03.122131 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46105
--builtin_ntp_servers=127.12.182.84:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:03:03.123155 30915 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46105
I20260928 03:03:03.123427 30915 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:03.124156 30915 heartbeater.cc:511] Master 127.12.182.126:46105 requested a full tablet report, sending...
I20260928 03:03:03.126143 30729 ts_manager.cc:194] Registered new tserver with Master: 6e3ffd679398410eb94e2e479d5c420b (127.12.182.65:39571)
I20260928 03:03:03.127561 30729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.65:43471
W20260928 03:03:03.434391 30919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:03.434648 30919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:03.434767 30919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:03.441859 30919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:03.441954 30919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:03.442061 30919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:03:03.450734 30919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:03.451867 30919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:03.453517 30919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:03.465749 30924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:03.465795 30925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:03.465721 30927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:03.466418 30919 server_base.cc:1034] running on GCE node
I20260928 03:03:03.466953 30919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:03.467767 30919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:03.468971 30919 hybrid_clock.cc:648] HybridClock initialized: now 1790564583468925 us; error 41 us; skew 500 ppm
I20260928 03:03:03.469256 30919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:03.471320 30919 webserver.cc:460] Webserver started at http://127.12.182.66:45299/ using document root <none> and password file <none>
I20260928 03:03:03.471951 30919 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:03.472051 30919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:03.472332 30919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:03.475025 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "1ffc8aad4847458399627116468119b7"
format_stamp: "Formatted at 2026-09-28 03:03:03 on dist-test-slave-k5rg"
server_key: "3e891263b382e82be1dd315ca321965e"
server_key_iv: "e3292a08411ad28cb9381a8a0a2afbf5"
server_key_version: "encryptionkey@0"
I20260928 03:03:03.475800 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "1ffc8aad4847458399627116468119b7"
format_stamp: "Formatted at 2026-09-28 03:03:03 on dist-test-slave-k5rg"
server_key: "3e891263b382e82be1dd315ca321965e"
server_key_iv: "e3292a08411ad28cb9381a8a0a2afbf5"
server_key_version: "encryptionkey@0"
I20260928 03:03:03.480470 30919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 03:03:03.483778 30933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:03.484997 30919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:03:03.485152 30919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "1ffc8aad4847458399627116468119b7"
format_stamp: "Formatted at 2026-09-28 03:03:03 on dist-test-slave-k5rg"
server_key: "3e891263b382e82be1dd315ca321965e"
server_key_iv: "e3292a08411ad28cb9381a8a0a2afbf5"
server_key_version: "encryptionkey@0"
I20260928 03:03:03.485343 30919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:03.502400 30919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:03.503188 30919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:03.503453 30919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:03.504392 30919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:03.505779 30919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:03.505875 30919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:03.505978 30919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:03.506047 30919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:03.535501 30919 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:39891
I20260928 03:03:03.535532 31045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:39891 every 8 connection(s)
I20260928 03:03:03.536798 30919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:03:03.544992 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 30919
I20260928 03:03:03.545152 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 03:03:03.545620 13017 external_mini_cluster.cc:1428] Setting key 14a3384999a8c201cbf71b76890bbc74
I20260928 03:03:03.548367 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46105
--builtin_ntp_servers=127.12.182.84:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:03:03.548856 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:03.539237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:54233 (local address 127.12.182.126:46105)
0928 03:03:03.539405 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:03.539409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:03.540727 (+ 1318us) server_negotiation.cc:407] Connection header received
0928 03:03:03.542673 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:03.542679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:03.542761 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:03.542846 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:03.543857 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.544544 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.546190 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.546610 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.546713 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:03.548112 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:03.548119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:03.548123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:03.548141 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:03.548173 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:03.548184 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:03.548260 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:03.548452 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 03:03:03.548548 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 03:03:03.549870 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:03.539754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46105 (local address 127.12.182.66:54233)
0928 03:03:03.540585 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:03.540626 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:03.542426 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:03.543063 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:03.543070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:03.543150 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:03.543690 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:03.543707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.544693 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.544699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:03.546051 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:03.546064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.546827 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.546833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:03.547105 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:03.547835 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:03:03.547838 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:03.547873 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:03:03.547941 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:03.548265 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:03.548275 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:03.548282 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 03:03:03.548429 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 03:03:03.548799 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":251,"threads_started":1}
I20260928 03:03:03.551045 31046 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46105
I20260928 03:03:03.551309 31046 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:03.551993 31046 heartbeater.cc:511] Master 127.12.182.126:46105 requested a full tablet report, sending...
I20260928 03:03:03.553326 30729 ts_manager.cc:194] Registered new tserver with Master: 1ffc8aad4847458399627116468119b7 (127.12.182.66:39891)
I20260928 03:03:03.554320 30729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.66:54233
W20260928 03:03:03.856501 31049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:03.856824 31049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:03.856921 31049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:03.864153 31049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:03.864244 31049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:03.864358 31049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:03:03.873240 31049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:46105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:03.874442 31049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:03.876077 31049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:03.888156 31057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:03.888234 31054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:03.888208 31055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:03.888906 31049 server_base.cc:1034] running on GCE node
I20260928 03:03:03.889506 31049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:03.890414 31049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:03.891609 31049 hybrid_clock.cc:648] HybridClock initialized: now 1790564583891547 us; error 57 us; skew 500 ppm
I20260928 03:03:03.891904 31049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:03.894066 31049 webserver.cc:460] Webserver started at http://127.12.182.67:45755/ using document root <none> and password file <none>
I20260928 03:03:03.894711 31049 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:03.894795 31049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:03.895056 31049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:03.897845 31049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "89e37571a2644288973ec8a60838f759"
format_stamp: "Formatted at 2026-09-28 03:03:03 on dist-test-slave-k5rg"
server_key: "a424b151c32d405c36db5e2dad181afa"
server_key_iv: "b071d8fb012ffe95f21b652b33014a0d"
server_key_version: "encryptionkey@0"
I20260928 03:03:03.898568 31049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "89e37571a2644288973ec8a60838f759"
format_stamp: "Formatted at 2026-09-28 03:03:03 on dist-test-slave-k5rg"
server_key: "a424b151c32d405c36db5e2dad181afa"
server_key_iv: "b071d8fb012ffe95f21b652b33014a0d"
server_key_version: "encryptionkey@0"
I20260928 03:03:03.903753 31049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:03:03.907150 31063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:03.908478 31049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:03:03.908634 31049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "89e37571a2644288973ec8a60838f759"
format_stamp: "Formatted at 2026-09-28 03:03:03 on dist-test-slave-k5rg"
server_key: "a424b151c32d405c36db5e2dad181afa"
server_key_iv: "b071d8fb012ffe95f21b652b33014a0d"
server_key_version: "encryptionkey@0"
I20260928 03:03:03.908824 31049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:03.921705 31049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:03.922423 31049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:03.922736 31049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:03.923638 31049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:03.925002 31049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:03.925088 31049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:03.925168 31049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:03.925220 31049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:03.954493 31049 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:34425
I20260928 03:03:03.954535 31175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:34425 every 8 connection(s)
I20260928 03:03:03.955783 31049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:03:03.960594 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 31049
I20260928 03:03:03.960780 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 03:03:03.961301 13017 external_mini_cluster.cc:1428] Setting key 8e0e9b7be9076a761cf17407873230d0
I20260928 03:03:03.967675 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:03.958086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:56659 (local address 127.12.182.126:46105)
0928 03:03:03.958284 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:03.958289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:03.959517 (+ 1228us) server_negotiation.cc:407] Connection header received
0928 03:03:03.961640 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:03.961647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:03.961729 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:03.961817 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:03.962938 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.963765 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.965247 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.965700 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.965813 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:03.966980 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:03.966985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:03.966988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:03.967005 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:03.967038 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:03.967051 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:03.967121 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:03.967353 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 03:03:03.967433 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:03:03.968612 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:03.958565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:46105 (local address 127.12.182.67:56659)
0928 03:03:03.959377 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:03.959411 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:03.961338 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:03.962051 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:03.962059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:03.962140 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:03.962755 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:03.962779 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.963913 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.963919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:03.965099 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:03.965111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.965881 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.965885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:03.966073 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:03.966658 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:03:03.966660 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:03.966697 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:03:03.966768 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:03.967172 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:03.967181 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:03.967188 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 03:03:03.967341 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 03:03:03.967623 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":242,"threads_started":1}
I20260928 03:03:03.969889 31176 heartbeater.cc:347] Connected to a master server at 127.12.182.126:46105
I20260928 03:03:03.970156 31176 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:03.970901 31176 heartbeater.cc:511] Master 127.12.182.126:46105 requested a full tablet report, sending...
I20260928 03:03:03.972343 30729 ts_manager.cc:194] Registered new tserver with Master: 89e37571a2644288973ec8a60838f759 (127.12.182.67:34425)
I20260928 03:03:03.973100 30729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.67:56659
I20260928 03:03:03.976191 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:03:03.985307 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:03.980067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.12.182.126:46105)
0928 03:03:03.980310 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:03.980315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:03.980589 (+ 274us) server_negotiation.cc:407] Connection header received
0928 03:03:03.980766 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:03.980770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:03.980840 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:03.980916 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:03.981333 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.982060 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.983377 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:03.983784 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:03.983874 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:03.984720 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:03.984725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:03.984728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:03.984745 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:03.984776 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:03.984787 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:03.984852 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:03.984933 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 03:03:03.985003 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 03:03:03.988873 30729 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:03:03.995659 30729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:03:04.019248 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.008289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:39891 (local address 127.0.0.1:55508)
0928 03:03:04.009243 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.009300 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:04.009503 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:04.010795 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:04.010804 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:04.010838 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:04.011256 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:04.011271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.013302 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.013308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:04.015758 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:04.015775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.018075 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.018081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:04.018307 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.018369 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 03:03:04.018507 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 03:03:04.018728 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":2076,"threads_started":1}
I20260928 03:03:04.019256 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.008984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.12.182.66:39891)
0928 03:03:04.010094 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:04.010104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:04.010138 (+ 34us) server_negotiation.cc:407] Connection header received
0928 03:03:04.010244 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:04.010252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:04.010514 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:04.010708 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:04.011491 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.013146 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.015987 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.017417 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.017529 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.017895 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:04.018578 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 03:03:04.018854 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":843,"threads_started":1}
I20260928 03:03:04.021544 31189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.008351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39571 (local address 127.0.0.1:51126)
0928 03:03:04.010596 (+ 2245us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.010611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:04.010816 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:04.013148 (+ 2332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:04.013153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:04.013180 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:04.013651 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:04.013665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.016736 (+ 3071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.016741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:04.018907 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:04.018921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.020900 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.020906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:04.021069 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.021102 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 03:03:04.021187 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 03:03:04.021327 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2145,"thread_start_us":808,"threads_started":1}
I20260928 03:03:04.021807 31191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.009011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.12.182.65:39571)
0928 03:03:04.010262 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:04.010268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:04.010706 (+ 438us) server_negotiation.cc:407] Connection header received
0928 03:03:04.011143 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:04.011151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:04.011425 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:04.011597 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:04.013849 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.015331 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.019140 (+ 3809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.020415 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.020518 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.020740 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:04.021289 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 03:03:04.021462 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1110,"thread_start_us":1015,"threads_started":1}
I20260928 03:03:04.023633 30981 tablet_service.cc:1467] Processing CreateTablet for tablet 358ecca6caaf4a28848ddb7d78c13519 (DEFAULT_TABLE table=test-table [id=98b0ffe23de646a2836e519abca798b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:04.024873 30981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358ecca6caaf4a28848ddb7d78c13519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:04.026892 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.010743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:34425 (local address 127.0.0.1:58268)
0928 03:03:04.017927 (+ 7184us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.017946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:04.018084 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:04.018969 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:04.018975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:04.019002 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:04.019964 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:04.019978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.022680 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.022684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:04.024027 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:04.024040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.026095 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.026101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:04.026317 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.026351 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 03:03:04.026446 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 03:03:04.026574 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6684,"thread_start_us":6531,"threads_started":1}
I20260928 03:03:04.027222 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.011250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.12.182.67:34425)
0928 03:03:04.017931 (+ 6681us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:04.017939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:04.018048 (+ 109us) server_negotiation.cc:407] Connection header received
0928 03:03:04.018247 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:04.018254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:04.018465 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:04.018678 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:04.020181 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.022538 (+ 2357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.024207 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.025550 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.025713 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.025974 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:04.026514 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 03:03:04.026683 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6521,"thread_start_us":6371,"threads_started":1}
I20260928 03:03:04.026895 30850 tablet_service.cc:1467] Processing CreateTablet for tablet 358ecca6caaf4a28848ddb7d78c13519 (DEFAULT_TABLE table=test-table [id=98b0ffe23de646a2836e519abca798b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:04.028066 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358ecca6caaf4a28848ddb7d78c13519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:04.033406 31111 tablet_service.cc:1467] Processing CreateTablet for tablet 358ecca6caaf4a28848ddb7d78c13519 (DEFAULT_TABLE table=test-table [id=98b0ffe23de646a2836e519abca798b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:04.034899 31111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358ecca6caaf4a28848ddb7d78c13519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:04.038044 31194 tablet_bootstrap.cc:492] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: Bootstrap starting.
I20260928 03:03:04.040252 31195 tablet_bootstrap.cc:492] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b: Bootstrap starting.
I20260928 03:03:04.042627 31194 tablet_bootstrap.cc:654] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:04.043555 31195 tablet_bootstrap.cc:654] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:04.043562 31194 log.cc:824] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:04.044757 31195 log.cc:824] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:04.046319 31194 tablet_bootstrap.cc:492] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: No bootstrap required, opened a new log
I20260928 03:03:04.046566 31194 ts_tablet_manager.cc:1397] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 03:03:04.047729 31195 tablet_bootstrap.cc:492] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b: No bootstrap required, opened a new log
I20260928 03:03:04.047986 31195 ts_tablet_manager.cc:1397] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 03:03:04.049256 31198 tablet_bootstrap.cc:492] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: Bootstrap starting.
I20260928 03:03:04.052798 31198 tablet_bootstrap.cc:654] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:04.054024 31198 log.cc:824] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:04.055665 31195 raft_consensus.cc:348] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffc8aad4847458399627116468119b7" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 39891 } } peers { permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 } } peers { permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 } }
I20260928 03:03:04.056115 31195 raft_consensus.cc:374] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:04.056216 31195 raft_consensus.cc:729] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3ffd679398410eb94e2e479d5c420b, State: Initialized, Role: FOLLOWER
I20260928 03:03:04.056419 31194 raft_consensus.cc:348] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffc8aad4847458399627116468119b7" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 39891 } } peers { permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 } } peers { permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 } }
I20260928 03:03:04.056789 31194 raft_consensus.cc:374] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:04.056874 31194 raft_consensus.cc:729] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffc8aad4847458399627116468119b7, State: Initialized, Role: FOLLOWER
I20260928 03:03:04.056841 31195 consensus_queue.cc:260] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffc8aad4847458399627116468119b7" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 39891 } } peers { permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 } } peers { permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 } }
I20260928 03:03:04.057175 31198 tablet_bootstrap.cc:492] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: No bootstrap required, opened a new log
I20260928 03:03:04.057447 31198 ts_tablet_manager.cc:1397] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260928 03:03:04.057389 31194 consensus_queue.cc:260] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffc8aad4847458399627116468119b7" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 39891 } } peers { permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 } } peers { permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 } }
I20260928 03:03:04.059006 31194 ts_tablet_manager.cc:1428] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 03:03:04.059449 31046 heartbeater.cc:503] Master 127.12.182.126:46105 was elected leader, sending a full tablet report...
I20260928 03:03:04.063064 31201 raft_consensus.cc:482] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:03:04.063393 31201 raft_consensus.cc:504] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffc8aad4847458399627116468119b7" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 39891 } } peers { permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 } } peers { permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 } }
I20260928 03:03:04.065119 31201 leader_election.cc:288] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e3ffd679398410eb94e2e479d5c420b (127.12.182.65:39571), 89e37571a2644288973ec8a60838f759 (127.12.182.67:34425)
I20260928 03:03:04.068540 31198 raft_consensus.cc:348] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffc8aad4847458399627116468119b7" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 39891 } } peers { permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 } } peers { permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 } }
I20260928 03:03:04.069051 31198 raft_consensus.cc:374] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:04.069155 31198 raft_consensus.cc:729] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e37571a2644288973ec8a60838f759, State: Initialized, Role: FOLLOWER
I20260928 03:03:04.069883 31198 consensus_queue.cc:260] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffc8aad4847458399627116468119b7" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 39891 } } peers { permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 } } peers { permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 } }
I20260928 03:03:04.071177 30915 heartbeater.cc:503] Master 127.12.182.126:46105 was elected leader, sending a full tablet report...
I20260928 03:03:04.071929 31195 ts_tablet_manager.cc:1428] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b: Time spent starting tablet: real 0.024s user 0.010s sys 0.010s
I20260928 03:03:04.073243 31176 heartbeater.cc:503] Master 127.12.182.126:46105 was elected leader, sending a full tablet report...
I20260928 03:03:04.073861 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.065848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:48899 (local address 127.12.182.67:34425)
0928 03:03:04.066036 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:04.066042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:04.066501 (+ 459us) server_negotiation.cc:407] Connection header received
0928 03:03:04.066671 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:04.066677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:04.066742 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:04.066819 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:04.067294 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.068069 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.070466 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.071595 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.071733 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.071806 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:04.073445 (+ 1639us) server_negotiation.cc:299] Negotiation successful
0928 03:03:04.073584 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 03:03:04.074539 31202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.065343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:34425 (local address 127.12.182.66:48899)
0928 03:03:04.066404 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.066419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:04.066542 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:04.066864 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:04.066868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:04.066892 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:04.067166 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:04.067174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.068188 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.068194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:04.070204 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:04.070224 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.072653 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.072658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:04.072852 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.073294 (+ 442us) client_negotiation.cc:769] Sending connection context
0928 03:03:04.074070 (+ 776us) client_negotiation.cc:240] Negotiation successful
0928 03:03:04.074228 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":784,"threads_started":1}
I20260928 03:03:04.075475 31131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358ecca6caaf4a28848ddb7d78c13519" candidate_uuid: "1ffc8aad4847458399627116468119b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e37571a2644288973ec8a60838f759" is_pre_election: true
I20260928 03:03:04.076005 31131 raft_consensus.cc:2454] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ffc8aad4847458399627116468119b7 in term 0.
I20260928 03:03:04.078298 31198 ts_tablet_manager.cc:1428] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: Time spent starting tablet: real 0.021s user 0.014s sys 0.000s
I20260928 03:03:04.078226 30937 leader_election.cc:302] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffc8aad4847458399627116468119b7, 89e37571a2644288973ec8a60838f759; no voters:
I20260928 03:03:04.078661 31201 raft_consensus.cc:2781] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:03:04.078799 31201 raft_consensus.cc:482] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:03:04.078891 31201 raft_consensus.cc:3037] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:04.081450 31201 raft_consensus.cc:504] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffc8aad4847458399627116468119b7" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 39891 } } peers { permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 } } peers { permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 } }
I20260928 03:03:04.082096 31201 leader_election.cc:288] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6e3ffd679398410eb94e2e479d5c420b (127.12.182.65:39571), 89e37571a2644288973ec8a60838f759 (127.12.182.67:34425)
I20260928 03:03:04.082594 31131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358ecca6caaf4a28848ddb7d78c13519" candidate_uuid: "1ffc8aad4847458399627116468119b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e37571a2644288973ec8a60838f759"
I20260928 03:03:04.082785 31131 raft_consensus.cc:3037] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:04.083845 31191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.065808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:43555 (local address 127.12.182.65:39571)
0928 03:03:04.065986 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:04.065992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:04.075313 (+ 9321us) server_negotiation.cc:407] Connection header received
0928 03:03:04.075565 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:04.075570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:04.075664 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:04.075725 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:04.079491 (+ 3766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.080298 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.081929 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.083031 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.083171 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.083220 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:04.083532 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 03:03:04.083633 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 03:03:04.083941 31202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.065399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39571 (local address 127.12.182.66:43555)
0928 03:03:04.075218 (+ 9819us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.075236 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:04.075413 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:04.075903 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:04.075907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:04.075933 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:04.076078 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:04.076088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.080499 (+ 4411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.080504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:04.081766 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:04.081781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.083193 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.083200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:04.083382 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.083417 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 03:03:04.083503 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 03:03:04.083659 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9695,"thread_start_us":12076,"threads_started":1}
I20260928 03:03:04.084918 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358ecca6caaf4a28848ddb7d78c13519" candidate_uuid: "1ffc8aad4847458399627116468119b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3ffd679398410eb94e2e479d5c420b"
I20260928 03:03:04.084883 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358ecca6caaf4a28848ddb7d78c13519" candidate_uuid: "1ffc8aad4847458399627116468119b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3ffd679398410eb94e2e479d5c420b" is_pre_election: true
I20260928 03:03:04.085458 31131 raft_consensus.cc:2454] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ffc8aad4847458399627116468119b7 in term 1.
I20260928 03:03:04.085450 30870 raft_consensus.cc:2454] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ffc8aad4847458399627116468119b7 in term 0.
I20260928 03:03:04.085832 30937 leader_election.cc:302] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffc8aad4847458399627116468119b7, 89e37571a2644288973ec8a60838f759; no voters:
I20260928 03:03:04.086086 31201 raft_consensus.cc:2781] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:04.086575 31201 raft_consensus.cc:686] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 1 LEADER]: Becoming Leader. State: Replica: 1ffc8aad4847458399627116468119b7, State: Running, Role: LEADER
I20260928 03:03:04.087014 31201 consensus_queue.cc:237] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffc8aad4847458399627116468119b7" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 39891 } } peers { permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 } } peers { permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 } }
I20260928 03:03:04.091240 30728 catalog_manager.cc:5495] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ffc8aad4847458399627116468119b7 (127.12.182.66). New cstate: current_term: 1 leader_uuid: "1ffc8aad4847458399627116468119b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffc8aad4847458399627116468119b7" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 39891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 } health_report { overall_health: UNKNOWN } } }
W20260928 03:03:04.110809 30916 tablet.cc:2369] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:03:04.120280 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.115221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55524 (local address 127.12.182.66:39891)
0928 03:03:04.115412 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:04.115417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:04.115436 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:03:04.115515 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:04.115518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:04.115601 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:04.115727 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:04.116071 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.116761 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.118311 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.118725 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.118853 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.119313 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:04.119745 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:04.119872 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:04.119971 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 03:03:04.120047 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 03:03:04.127897 30870 raft_consensus.cc:3037] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:04.128113 31131 raft_consensus.cc:1261] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759 [term 1 FOLLOWER]: Refusing update from remote peer 1ffc8aad4847458399627116468119b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:04.128938 31205 consensus_queue.cc:1035] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e37571a2644288973ec8a60838f759" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:04.130584 30870 raft_consensus.cc:1261] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b [term 1 FOLLOWER]: Refusing update from remote peer 1ffc8aad4847458399627116468119b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:04.131326 31205 consensus_queue.cc:1035] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3ffd679398410eb94e2e479d5c420b" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:04.138314 31208 mvcc.cc:205] Tried to move back new op lower bound from 7334152536578891776 to 7334152536423833600. Current Snapshot: MvccSnapshot[applied={T|T < 7334152536578891776}]
I20260928 03:03:04.163110 30728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:04.163406 30728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:04.166944 30728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03a5ccc577984ce5a5fafae44a41a6a7: Sending DeleteTablet for 3 replicas of tablet 358ecca6caaf4a28848ddb7d78c13519
I20260928 03:03:04.168073 30981 tablet_service.cc:1514] Processing DeleteTablet for tablet 358ecca6caaf4a28848ddb7d78c13519 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:04 UTC) from {username='slave'} at 127.0.0.1:55508
I20260928 03:03:04.168073 30850 tablet_service.cc:1514] Processing DeleteTablet for tablet 358ecca6caaf4a28848ddb7d78c13519 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:04 UTC) from {username='slave'} at 127.0.0.1:51126
I20260928 03:03:04.168118 31111 tablet_service.cc:1514] Processing DeleteTablet for tablet 358ecca6caaf4a28848ddb7d78c13519 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:04 UTC) from {username='slave'} at 127.0.0.1:58268
I20260928 03:03:04.169287 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 30788
I20260928 03:03:04.186652 31220 tablet_replica.cc:330] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: stopping tablet replica
I20260928 03:03:04.187207 31220 raft_consensus.cc:2229] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:04.187712 31220 raft_consensus.cc:2258] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:04.189891 31191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.170385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.12.182.65:39571)
0928 03:03:04.170551 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:04.170556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:04.181378 (+ 10822us) server_negotiation.cc:407] Connection header received
0928 03:03:04.181553 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:04.181559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:04.181753 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:04.181857 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:04.182490 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.183760 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.185762 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.186444 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.186540 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.188033 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:04.188045 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:04.188062 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:04.188855 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:04.188916 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:04.188940 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:04.189023 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:04.189304 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 03:03:04.189421 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:03:04.190199 31220 ts_tablet_manager.cc:1905] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:04.191100 30910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:04.192549 31218 tablet_replica.cc:330] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: stopping tablet replica
I20260928 03:03:04.193070 31218 raft_consensus.cc:2229] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:03:04.193714 31218 raft_consensus.cc:2258] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:04.199060 31218 ts_tablet_manager.cc:1905] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:04.202929 31220 ts_tablet_manager.cc:1918] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:04.203096 31220 log.cc:1196] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/358ecca6caaf4a28848ddb7d78c13519
I20260928 03:03:04.203636 31220 ts_tablet_manager.cc:1939] T 358ecca6caaf4a28848ddb7d78c13519 P 89e37571a2644288973ec8a60838f759: Deleting consensus metadata
I20260928 03:03:04.205183 31218 ts_tablet_manager.cc:1918] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:04.205332 31218 log.cc:1196] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/358ecca6caaf4a28848ddb7d78c13519
I20260928 03:03:04.205551 30716 catalog_manager.cc:4841] TS 89e37571a2644288973ec8a60838f759 (127.12.182.67:34425): tablet 358ecca6caaf4a28848ddb7d78c13519 (table test-table [id=98b0ffe23de646a2836e519abca798b4]) successfully deleted
I20260928 03:03:04.205816 31218 ts_tablet_manager.cc:1939] T 358ecca6caaf4a28848ddb7d78c13519 P 1ffc8aad4847458399627116468119b7: Deleting consensus metadata
I20260928 03:03:04.207410 30716 catalog_manager.cc:4841] TS 1ffc8aad4847458399627116468119b7 (127.12.182.66:39891): tablet 358ecca6caaf4a28848ddb7d78c13519 (table test-table [id=98b0ffe23de646a2836e519abca798b4]) successfully deleted
I20260928 03:03:04.287429 31219 tablet_replica.cc:330] T 358ecca6caaf4a28848ddb7d78c13519 P 6e3ffd679398410eb94e2e479d5c420b: stopping tablet replica
I20260928 03:03:04.287657 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 30788
I20260928 03:03:04.302624 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 30919
W20260928 03:03:04.302779 30715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.182.65:39571 (error 108)
I20260928 03:03:04.305074 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.304557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39571 (local address 127.0.0.1:51152)
0928 03:03:04.304759 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.304896 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 03:03:04.305359 30715 catalog_manager.cc:4584] TS 6e3ffd679398410eb94e2e479d5c420b (127.12.182.65:39571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 358ecca6caaf4a28848ddb7d78c13519: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39571: connect: Connection refused (error 111)
I20260928 03:03:04.309773 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.303418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.12.182.66:39891)
0928 03:03:04.303585 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:04.303589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:04.303607 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:03:04.303843 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:04.303848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:04.303956 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:04.304068 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:04.304563 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.305485 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.306780 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.307229 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.307346 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.308377 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:04.308386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:04.308400 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:04.308981 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:04.309030 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:04.309052 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:04.309151 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:04.309290 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 03:03:04.309392 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:03:04.310520 31041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:04.359171 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.358607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39571 (local address 127.0.0.1:51156)
0928 03:03:04.358860 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.358973 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 03:03:04.360395 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.360004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39571 (local address 127.0.0.1:51168)
0928 03:03:04.360167 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.360251 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 03:03:04.420445 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 30919
I20260928 03:03:04.421002 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.420423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39571 (local address 127.0.0.1:51172)
0928 03:03:04.420723 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.420821 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260928 03:03:04.422133 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.421691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39571 (local address 127.0.0.1:51174)
0928 03:03:04.421898 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.421976 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 03:03:04.435626 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 31049
I20260928 03:03:04.442144 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.436445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.12.182.67:34425)
0928 03:03:04.436724 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:04.436730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:04.436748 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:03:04.436807 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:04.436810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:04.436885 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:04.436968 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:04.437356 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.438137 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.439356 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:04.439747 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:04.439853 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:04.440834 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:04.440843 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:04.440857 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:04.441421 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:04.441468 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:04.441488 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:04.441592 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:04.441731 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 03:03:04.441834 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260928 03:03:04.442775 31171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:04.503067 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.502595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39571 (local address 127.0.0.1:51190)
0928 03:03:04.502813 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.502900 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 03:03:04.504072 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:04.503679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:39571 (local address 127.0.0.1:51198)
0928 03:03:04.503892 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:04.503962 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 03:03:04.533166 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 31049
I20260928 03:03:04.546180 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 30698
I20260928 03:03:04.547202 30759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:04.654804 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 30698
2026-09-28T03:03:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2563 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:04 dist-test-slave-k5rg krb5kdc[31229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:04 dist-test-slave-k5rg krb5kdc[31229](info): set up 2 sockets
Sep 28 03:03:04 dist-test-slave-k5rg krb5kdc[31229](info): commencing operation
krb5kdc: starting...
W20260928 03:03:05.673653 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:05 dist-test-slave-k5rg krb5kdc[31229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:03:05.910066 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41793
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:41793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:06.232981 31242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:06.233258 31242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:06.233389 31242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:06.240818 31242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:06.240931 31242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:06.241004 31242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:06.241084 31242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:06.241156 31242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:06.250999 31242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:41793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:06.252276 31242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:06.253518 31242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:06.264838 31247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:06.264809 31250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:06.264796 31248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:06.265398 31242 server_base.cc:1034] running on GCE node
I20260928 03:03:06.266130 31242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:06.267423 31242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:06.268606 31242 hybrid_clock.cc:648] HybridClock initialized: now 1790564586268557 us; error 42 us; skew 500 ppm
I20260928 03:03:06.268893 31242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:06 dist-test-slave-k5rg krb5kdc[31229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564586, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:06.271780 31242 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:03:06.273129 31242 webserver.cc:460] Webserver started at http://127.12.182.126:40535/ using document root <none> and password file <none>
I20260928 03:03:06.273828 31242 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:06.273942 31242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:06.274226 31242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:06.277053 31242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "8cab689fab58497382fcac54284a8420"
format_stamp: "Formatted at 2026-09-28 03:03:06 on dist-test-slave-k5rg"
server_key: "f207cd33b950ccb0bc0954e1ef85a732"
server_key_iv: "33ecf8ac77e5a66e0010b22f36d833cd"
server_key_version: "encryptionkey@0"
I20260928 03:03:06.277814 31242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "8cab689fab58497382fcac54284a8420"
format_stamp: "Formatted at 2026-09-28 03:03:06 on dist-test-slave-k5rg"
server_key: "f207cd33b950ccb0bc0954e1ef85a732"
server_key_iv: "33ecf8ac77e5a66e0010b22f36d833cd"
server_key_version: "encryptionkey@0"
I20260928 03:03:06.282618 31242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:03:06.285881 31257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:06.287158 31242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:03:06.287333 31242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "8cab689fab58497382fcac54284a8420"
format_stamp: "Formatted at 2026-09-28 03:03:06 on dist-test-slave-k5rg"
server_key: "f207cd33b950ccb0bc0954e1ef85a732"
server_key_iv: "33ecf8ac77e5a66e0010b22f36d833cd"
server_key_version: "encryptionkey@0"
I20260928 03:03:06.287509 31242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:06.315912 31242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:06.320210 31242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:06.320478 31242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:06.337234 31242 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:41793
I20260928 03:03:06.337241 31308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:41793 every 8 connection(s)
I20260928 03:03:06.338579 31242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:06.342278 31309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:06.342391 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 31242
I20260928 03:03:06.342573 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:06.342999 13017 external_mini_cluster.cc:1428] Setting key d82de719937ae69a96237ecbc5af8d18
I20260928 03:03:06.356351 31309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420: Bootstrap starting.
Sep 28 03:03:06 dist-test-slave-k5rg krb5kdc[31229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:06.359831 31309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:06.360863 31309 log.cc:824] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:06.363938 31309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420: No bootstrap required, opened a new log
I20260928 03:03:06.365744 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:06.344757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.12.182.126:41793)
0928 03:03:06.345553 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:06.345564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:06.345629 (+ 65us) server_negotiation.cc:407] Connection header received
0928 03:03:06.347152 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:06.347181 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:06.347586 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:06.347947 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:06.349720 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:06.350974 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:06.352837 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:06.353592 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:06.353810 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:06.358549 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:06.358557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:06.358571 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:06.358611 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:06.361668 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:06.362354 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:06.362361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:06.362368 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:06.362521 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:06.362977 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:06.362981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:06.362983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:06.363531 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:06.363767 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:06.364296 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 03:03:06.364693 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":355,"threads_started":1}
I20260928 03:03:06.371832 31309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cab689fab58497382fcac54284a8420" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41793 } }
I20260928 03:03:06.372219 31309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:06.372366 31309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cab689fab58497382fcac54284a8420, State: Initialized, Role: FOLLOWER
I20260928 03:03:06.373015 31309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cab689fab58497382fcac54284a8420" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41793 } }
I20260928 03:03:06.373252 31309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:06.373361 31309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:06.373486 31309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:06.375638 31309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cab689fab58497382fcac54284a8420" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41793 } }
I20260928 03:03:06.376186 31309 leader_election.cc:302] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cab689fab58497382fcac54284a8420; no voters:
I20260928 03:03:06.376745 31309 leader_election.cc:288] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:06.376878 31314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:06.377480 31314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [term 1 LEADER]: Becoming Leader. State: Replica: 8cab689fab58497382fcac54284a8420, State: Running, Role: LEADER
I20260928 03:03:06.378057 31314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cab689fab58497382fcac54284a8420" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41793 } }
I20260928 03:03:06.378731 31309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:06.380514 31315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cab689fab58497382fcac54284a8420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cab689fab58497382fcac54284a8420" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41793 } } }
I20260928 03:03:06.380890 31315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:06.381755 31319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:06.381541 31316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cab689fab58497382fcac54284a8420. Latest consensus state: current_term: 1 leader_uuid: "8cab689fab58497382fcac54284a8420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cab689fab58497382fcac54284a8420" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41793 } } }
I20260928 03:03:06.381954 31316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:06.386786 31319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:06.395429 31319 catalog_manager.cc:1347] Generated new cluster ID: edfdb82b94bd4c0283bfeb2d61232fe2
I20260928 03:03:06.395532 31319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:06.411993 31319 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:06.413311 31319 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:06.422374 31319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420: Generated new TSK 0
I20260928 03:03:06.423171 31319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 03:03:06.475103 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41793
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:06.796350 31335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:06.796617 31335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:06.796746 31335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:06.804104 31335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:06.804225 31335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:06.804386 31335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:03:06.814234 31335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:06.815495 31335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:06.816859 31335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:06.829154 31341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:06.829187 31343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:06.829213 31340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:06.830027 31335 server_base.cc:1034] running on GCE node
I20260928 03:03:06.830593 31335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:06.831518 31335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:06.832712 31335 hybrid_clock.cc:648] HybridClock initialized: now 1790564586832662 us; error 39 us; skew 500 ppm
I20260928 03:03:06.833030 31335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:06 dist-test-slave-k5rg krb5kdc[31229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564586, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:06.836076 31335 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:03:06.837496 31335 webserver.cc:460] Webserver started at http://127.12.182.65:45463/ using document root <none> and password file <none>
I20260928 03:03:06.838196 31335 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:06.838315 31335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:06.838609 31335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:06.841410 31335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "2271114dfc8d47cb921a2e15583e8c88"
format_stamp: "Formatted at 2026-09-28 03:03:06 on dist-test-slave-k5rg"
server_key: "e7bbb94dc0db937a45bc170205cc9f3b"
server_key_iv: "df67561ba67144664b60465d2d957792"
server_key_version: "encryptionkey@0"
I20260928 03:03:06.842248 31335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "2271114dfc8d47cb921a2e15583e8c88"
format_stamp: "Formatted at 2026-09-28 03:03:06 on dist-test-slave-k5rg"
server_key: "e7bbb94dc0db937a45bc170205cc9f3b"
server_key_iv: "df67561ba67144664b60465d2d957792"
server_key_version: "encryptionkey@0"
I20260928 03:03:06.847385 31335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:03:06.850873 31350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:06.852272 31335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 03:03:06.852432 31335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "2271114dfc8d47cb921a2e15583e8c88"
format_stamp: "Formatted at 2026-09-28 03:03:06 on dist-test-slave-k5rg"
server_key: "e7bbb94dc0db937a45bc170205cc9f3b"
server_key_iv: "df67561ba67144664b60465d2d957792"
server_key_version: "encryptionkey@0"
I20260928 03:03:06.852625 31335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:06.866101 31335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:06.869946 31335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:06.870174 31335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:06.871034 31335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:06.872256 31335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:06.872327 31335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:06.872417 31335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:06.872462 31335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:06.902405 31335 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:44721
I20260928 03:03:06.902448 31462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:44721 every 8 connection(s)
I20260928 03:03:06.903659 31335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:03:06.907658 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 31335
I20260928 03:03:06.907867 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 03:03:06.908432 13017 external_mini_cluster.cc:1428] Setting key cd919367eaf1b9506f963d282fe6b511
Sep 28 03:03:06 dist-test-slave-k5rg krb5kdc[31229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564586, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:06.925407 31465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:06.906079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:57899 (local address 127.12.182.126:41793)
0928 03:03:06.906501 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:06.906506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:06.907502 (+ 996us) server_negotiation.cc:407] Connection header received
0928 03:03:06.910039 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:06.910044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:06.910118 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:06.910210 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:06.912316 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:06.913066 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:06.915043 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:06.915438 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:06.915530 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:06.919370 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:06.919375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:06.919379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:06.919419 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:06.921923 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:06.922916 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:06.922920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:06.922922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:06.923054 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:06.923575 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:06.923579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:06.923581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:06.923893 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:06.924031 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:06.924768 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 03:03:06.925002 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":207,"threads_started":1}
I20260928 03:03:06.926481 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:06.906591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41793 (local address 127.12.182.65:57899)
0928 03:03:06.907310 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:06.907370 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:06.909711 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:06.910510 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:06.910529 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:06.911301 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:06.912116 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:06.912141 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:06.913214 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:06.913219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:06.914907 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:06.914922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:06.915646 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:06.915653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:06.915925 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:06.916635 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:06.916663 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:06.919115 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:06.922132 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:06.922140 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:06.922155 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:06.922785 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:06.923175 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:06.923181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:06.923185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:06.923437 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:06.924039 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:06.924047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:06.924400 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 03:03:06.924639 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 03:03:06.925203 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":269,"threads_started":1}
I20260928 03:03:06.928182 31463 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41793
I20260928 03:03:06.928514 31463 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:06.929272 31463 heartbeater.cc:511] Master 127.12.182.126:41793 requested a full tablet report, sending...
I20260928 03:03:06.931665 31274 ts_manager.cc:194] Registered new tserver with Master: 2271114dfc8d47cb921a2e15583e8c88 (127.12.182.65:44721)
I20260928 03:03:06.933832 31274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:57899
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 03:03:06.965124 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41793
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:07.303009 31469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:07.303361 31469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:07.303493 31469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:07.311975 31469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:07.312139 31469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:07.312314 31469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:03:07.325433 31469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:07.326905 31469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:07.328279 31469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:07.346644 31474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:07.346660 31477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:07.347363 31475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:07.349252 31469 server_base.cc:1034] running on GCE node
I20260928 03:03:07.350071 31469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:07.351526 31469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:07.352785 31469 hybrid_clock.cc:648] HybridClock initialized: now 1790564587352571 us; error 213 us; skew 500 ppm
I20260928 03:03:07.353181 31469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:07 dist-test-slave-k5rg krb5kdc[31229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564587, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:07.357189 31469 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:03:07.359014 31469 webserver.cc:460] Webserver started at http://127.12.182.66:43643/ using document root <none> and password file <none>
I20260928 03:03:07.359717 31469 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:07.359834 31469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:07.360111 31469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:07.363529 31469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "76b698aa24d344d8a9613d8f4a9b3fb8"
format_stamp: "Formatted at 2026-09-28 03:03:07 on dist-test-slave-k5rg"
server_key: "4e77b1d81336b0c0abd86dc4e9e34f35"
server_key_iv: "f1779da39762695c7000f52847df5583"
server_key_version: "encryptionkey@0"
I20260928 03:03:07.364375 31469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "76b698aa24d344d8a9613d8f4a9b3fb8"
format_stamp: "Formatted at 2026-09-28 03:03:07 on dist-test-slave-k5rg"
server_key: "4e77b1d81336b0c0abd86dc4e9e34f35"
server_key_iv: "f1779da39762695c7000f52847df5583"
server_key_version: "encryptionkey@0"
I20260928 03:03:07.370875 31469 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 03:03:07.375146 31484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:07.376992 31469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 03:03:07.377172 31469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "76b698aa24d344d8a9613d8f4a9b3fb8"
format_stamp: "Formatted at 2026-09-28 03:03:07 on dist-test-slave-k5rg"
server_key: "4e77b1d81336b0c0abd86dc4e9e34f35"
server_key_iv: "f1779da39762695c7000f52847df5583"
server_key_version: "encryptionkey@0"
I20260928 03:03:07.377419 31469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:07.412609 31469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:07.417580 31469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:07.417933 31469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:07.419224 31469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:07.421218 31469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:07.421409 31469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:07.421743 31469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:07.421849 31469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:07.460440 31469 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:35101
I20260928 03:03:07.460487 31596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:35101 every 8 connection(s)
I20260928 03:03:07.461756 31469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:03:07.469388 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 31469
I20260928 03:03:07.469548 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 03:03:07.470098 13017 external_mini_cluster.cc:1428] Setting key 645d9bf2391c9aea81f247eec3c9651f
Sep 28 03:03:07 dist-test-slave-k5rg krb5kdc[31229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564587, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:07.481398 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:07.464257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:39211 (local address 127.12.182.126:41793)
0928 03:03:07.464724 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:07.464730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:07.465594 (+ 864us) server_negotiation.cc:407] Connection header received
0928 03:03:07.467353 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:07.467358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:07.467424 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:07.467517 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:07.468903 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:07.469685 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:07.471627 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:07.472126 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:07.472264 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:07.475810 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:07.475815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:07.475818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:07.475849 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:07.478341 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:07.479154 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:07.479158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:07.479160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:07.479326 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:07.479739 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:07.479742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:07.479744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:07.480077 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:07.480214 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:07.480832 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 03:03:07.481049 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":207,"threads_started":1}
I20260928 03:03:07.482327 31600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:07.464740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41793 (local address 127.12.182.66:39211)
0928 03:03:07.465440 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:07.465482 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:07.467122 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:07.467734 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:07.467742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:07.468240 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:07.468771 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:07.468788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:07.469859 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:07.469864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:07.471498 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:07.471512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:07.472340 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:07.472349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:07.472570 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:07.473208 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:07.473237 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:07.475598 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:07.478476 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:07.478485 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:07.478500 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:07.479060 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:07.479419 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:07.479423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:07.479427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:07.479649 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:07.480189 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:07.480195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:07.480499 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 03:03:07.480730 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 03:03:07.481137 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":261,"threads_started":1}
I20260928 03:03:07.483763 31597 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41793
I20260928 03:03:07.484086 31597 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:07.484892 31597 heartbeater.cc:511] Master 127.12.182.126:41793 requested a full tablet report, sending...
I20260928 03:03:07.486634 31274 ts_manager.cc:194] Registered new tserver with Master: 76b698aa24d344d8a9613d8f4a9b3fb8 (127.12.182.66:35101)
I20260928 03:03:07.487485 31274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:39211
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 03:03:07.523208 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41793
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:07.832569 31603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:07.832836 31603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:07.832957 31603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:07.840183 31603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:07.840288 31603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:07.840435 31603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:03:07.850328 31603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:41793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:07.851573 31603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:07.852833 31603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:07.865031 31608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:07.865293 31609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:07.865590 31611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:07.866078 31603 server_base.cc:1034] running on GCE node
I20260928 03:03:07.866659 31603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:07.867589 31603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:07.868798 31603 hybrid_clock.cc:648] HybridClock initialized: now 1790564587868750 us; error 34 us; skew 500 ppm
I20260928 03:03:07.869102 31603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:07 dist-test-slave-k5rg krb5kdc[31229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564587, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:07.872066 31603 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:03:07.873395 31603 webserver.cc:460] Webserver started at http://127.12.182.67:36419/ using document root <none> and password file <none>
I20260928 03:03:07.874168 31603 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:07.874274 31603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:07.874529 31603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:07.877362 31603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2"
format_stamp: "Formatted at 2026-09-28 03:03:07 on dist-test-slave-k5rg"
server_key: "77ad8b1ff73908790fdaa5458b490e75"
server_key_iv: "61792cb6f9a27c449e7f29302d777bb1"
server_key_version: "encryptionkey@0"
I20260928 03:03:07.878204 31603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2"
format_stamp: "Formatted at 2026-09-28 03:03:07 on dist-test-slave-k5rg"
server_key: "77ad8b1ff73908790fdaa5458b490e75"
server_key_iv: "61792cb6f9a27c449e7f29302d777bb1"
server_key_version: "encryptionkey@0"
I20260928 03:03:07.883263 31603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 03:03:07.886798 31618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:07.888116 31603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:03:07.888267 31603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2"
format_stamp: "Formatted at 2026-09-28 03:03:07 on dist-test-slave-k5rg"
server_key: "77ad8b1ff73908790fdaa5458b490e75"
server_key_iv: "61792cb6f9a27c449e7f29302d777bb1"
server_key_version: "encryptionkey@0"
I20260928 03:03:07.888450 31603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:07.903714 31603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:07.907730 31603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:07.907984 31603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:07.908887 31603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:07.910117 31603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:07.910197 31603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:07.910323 31603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:07.910380 31603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:07.937026 31463 heartbeater.cc:503] Master 127.12.182.126:41793 was elected leader, sending a full tablet report...
I20260928 03:03:07.939651 31603 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:46497
I20260928 03:03:07.939689 31730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:46497 every 8 connection(s)
I20260928 03:03:07.940862 31603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:03:07.945178 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 31603
I20260928 03:03:07.945336 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 03:03:07.945765 13017 external_mini_cluster.cc:1428] Setting key 5d87a135dd13225325f08f6fa163245f
Sep 28 03:03:07 dist-test-slave-k5rg krb5kdc[31229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564587, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:07.959555 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:07.943210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:33797 (local address 127.12.182.126:41793)
0928 03:03:07.943404 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:07.943408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:07.944489 (+ 1081us) server_negotiation.cc:407] Connection header received
0928 03:03:07.946564 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:07.946570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:07.946643 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:07.946731 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:07.948184 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:07.948904 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:07.950428 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:07.950841 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:07.950948 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:07.954223 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:07.954227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:07.954231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:07.954268 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:07.956691 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:07.957411 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:07.957415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:07.957418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:07.957532 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:07.957954 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:07.957957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:07.957959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:07.958229 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:07.958369 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:07.959020 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 03:03:07.959240 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 03:03:07.960636 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:07.943628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:41793 (local address 127.12.182.67:33797)
0928 03:03:07.944344 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:07.944396 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:07.946149 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:07.946986 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:07.946994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:07.947488 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:07.948028 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:07.948044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:07.949043 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:07.949047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:07.950297 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:07.950309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:07.951042 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:07.951050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:07.951267 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:07.951856 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:07.951878 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:07.954015 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:07.956854 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:07.956863 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:07.956879 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:07.957293 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:07.957652 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:07.957656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:07.957660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:07.957844 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:07.958347 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:07.958354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:07.958676 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 03:03:07.958900 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 03:03:07.959419 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":258,"threads_started":1}
I20260928 03:03:07.961982 31731 heartbeater.cc:347] Connected to a master server at 127.12.182.126:41793
I20260928 03:03:07.962260 31731 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:07.962988 31731 heartbeater.cc:511] Master 127.12.182.126:41793 requested a full tablet report, sending...
I20260928 03:03:07.964378 31274 ts_manager.cc:194] Registered new tserver with Master: 9b8cf135c4fa4e7ab8644a2ad63db7d2 (127.12.182.67:46497)
I20260928 03:03:07.965133 31274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:33797
I20260928 03:03:07.972401 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:03:07.988437 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:07.976182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.12.182.126:41793)
0928 03:03:07.976392 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:07.976399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:07.976727 (+ 328us) server_negotiation.cc:407] Connection header received
0928 03:03:07.976892 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:07.976896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:07.976956 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:07.977032 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:07.978374 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:07.979137 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:07.980460 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:07.980880 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:07.980974 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:07.983460 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:07.983465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:07.983469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:07.983499 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:07.986155 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:07.986814 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:07.986818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:07.986820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:07.986934 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:07.987266 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:07.987269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:07.987271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:07.987541 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:07.987760 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:07.987998 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 03:03:07.988124 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 03:03:07.992188 31274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:03:07.999045 31274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:03:08.045353 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.010482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.12.182.67:46497)
0928 03:03:08.037286 (+ 26804us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:08.037295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:08.037330 (+ 35us) server_negotiation.cc:407] Connection header received
0928 03:03:08.037430 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:08.037438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:08.037703 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:08.038812 (+ 1109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:08.039075 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.040100 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.043345 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.044239 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.044352 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.044784 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:08.044857 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 03:03:08.045010 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26654,"thread_start_us":26510,"threads_started":1}
I20260928 03:03:08.045442 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.010187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:46497 (local address 127.0.0.1:47456)
0928 03:03:08.036182 (+ 25995us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:08.036234 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:08.036460 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:08.037937 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:08.037944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:08.037978 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:08.038539 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:08.038555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.040362 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.040368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:08.043154 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:08.043170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.044410 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.044417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:08.044620 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.044683 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 03:03:08.044802 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 03:03:08.044991 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25687,"mutex_wait_us":387,"thread_start_us":27154,"threads_started":1}
I20260928 03:03:08.046061 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.010494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44721 (local address 127.0.0.1:53644)
0928 03:03:08.036824 (+ 26330us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:08.036844 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:08.037092 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:08.038060 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:08.038063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:08.038091 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:08.038696 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:08.038713 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.040642 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.040647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:08.043162 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:08.043172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.045202 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.045209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:08.045379 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.045412 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 03:03:08.045497 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 03:03:08.045640 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26192,"mutex_wait_us":136,"thread_start_us":26138,"threads_started":1}
I20260928 03:03:08.046653 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.011457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.12.182.66:35101)
0928 03:03:08.035255 (+ 23798us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:08.035267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:08.037870 (+ 2603us) server_negotiation.cc:407] Connection header received
0928 03:03:08.037962 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:08.037968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:08.038158 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:08.038357 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:08.039160 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.040765 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.042980 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.044288 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.044419 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.046166 (+ 1747us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:08.046230 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 03:03:08.046382 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23558,"thread_start_us":23814,"threads_started":1}
I20260928 03:03:08.046802 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.011144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.12.182.65:44721)
0928 03:03:08.035280 (+ 24136us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:08.035291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:08.036973 (+ 1682us) server_negotiation.cc:407] Connection header received
0928 03:03:08.037297 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:08.037305 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:08.037574 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:08.037763 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:08.038896 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.040504 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.043330 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.044714 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.044838 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.045052 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:08.046347 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0928 03:03:08.046503 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23920,"thread_start_us":24163,"threads_started":1}
I20260928 03:03:08.046501 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.010881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:35101 (local address 127.0.0.1:42566)
0928 03:03:08.037597 (+ 26716us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:08.037638 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:08.037753 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:08.038575 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:08.038579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:08.038602 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:08.039006 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:08.039018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.040926 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.040931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:08.042831 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:08.042843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.045781 (+ 2938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.045788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:08.045973 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.046008 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 03:03:08.046089 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 03:03:08.046208 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26602,"thread_start_us":24933,"threads_started":1}
I20260928 03:03:08.049808 31666 tablet_service.cc:1467] Processing CreateTablet for tablet 13bc887765be49548971f8ef2fd3bf96 (DEFAULT_TABLE table=test-table [id=1a73133a046c42eba80303c41194ca5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:08.050130 31532 tablet_service.cc:1467] Processing CreateTablet for tablet 13bc887765be49548971f8ef2fd3bf96 (DEFAULT_TABLE table=test-table [id=1a73133a046c42eba80303c41194ca5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:08.050256 31398 tablet_service.cc:1467] Processing CreateTablet for tablet 13bc887765be49548971f8ef2fd3bf96 (DEFAULT_TABLE table=test-table [id=1a73133a046c42eba80303c41194ca5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:08.051218 31532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bc887765be49548971f8ef2fd3bf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:08.051290 31666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bc887765be49548971f8ef2fd3bf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:08.051476 31398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bc887765be49548971f8ef2fd3bf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:08.062892 31749 tablet_bootstrap.cc:492] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: Bootstrap starting.
I20260928 03:03:08.065312 31750 tablet_bootstrap.cc:492] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: Bootstrap starting.
I20260928 03:03:08.066354 31751 tablet_bootstrap.cc:492] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: Bootstrap starting.
I20260928 03:03:08.066644 31749 tablet_bootstrap.cc:654] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:08.067955 31749 log.cc:824] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:08.068696 31750 tablet_bootstrap.cc:654] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:08.069293 31751 tablet_bootstrap.cc:654] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:08.069962 31750 log.cc:824] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:08.070389 31751 log.cc:824] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:08.072355 31749 tablet_bootstrap.cc:492] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: No bootstrap required, opened a new log
I20260928 03:03:08.072613 31751 tablet_bootstrap.cc:492] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: No bootstrap required, opened a new log
I20260928 03:03:08.072636 31749 ts_tablet_manager.cc:1397] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260928 03:03:08.072841 31751 ts_tablet_manager.cc:1397] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 03:03:08.080228 31750 tablet_bootstrap.cc:492] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: No bootstrap required, opened a new log
I20260928 03:03:08.080513 31750 ts_tablet_manager.cc:1397] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260928 03:03:08.080577 31751 raft_consensus.cc:348] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 46497 } } peers { permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 } } peers { permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 } }
I20260928 03:03:08.080942 31751 raft_consensus.cc:374] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:08.081037 31751 raft_consensus.cc:729] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2271114dfc8d47cb921a2e15583e8c88, State: Initialized, Role: FOLLOWER
I20260928 03:03:08.081595 31751 consensus_queue.cc:260] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 46497 } } peers { permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 } } peers { permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 } }
I20260928 03:03:08.083241 31751 ts_tablet_manager.cc:1428] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260928 03:03:08.083256 31749 raft_consensus.cc:348] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 46497 } } peers { permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 } } peers { permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 } }
I20260928 03:03:08.083731 31749 raft_consensus.cc:374] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:08.083875 31749 raft_consensus.cc:729] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b698aa24d344d8a9613d8f4a9b3fb8, State: Initialized, Role: FOLLOWER
I20260928 03:03:08.084648 31749 consensus_queue.cc:260] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 46497 } } peers { permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 } } peers { permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 } }
I20260928 03:03:08.091777 31597 heartbeater.cc:503] Master 127.12.182.126:41793 was elected leader, sending a full tablet report...
I20260928 03:03:08.092839 31749 ts_tablet_manager.cc:1428] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: Time spent starting tablet: real 0.020s user 0.010s sys 0.010s
I20260928 03:03:08.092777 31750 raft_consensus.cc:348] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 46497 } } peers { permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 } } peers { permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 } }
I20260928 03:03:08.093240 31750 raft_consensus.cc:374] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:08.093341 31750 raft_consensus.cc:729] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8cf135c4fa4e7ab8644a2ad63db7d2, State: Initialized, Role: FOLLOWER
I20260928 03:03:08.093900 31750 consensus_queue.cc:260] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 46497 } } peers { permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 } } peers { permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 } }
I20260928 03:03:08.094901 31731 heartbeater.cc:503] Master 127.12.182.126:41793 was elected leader, sending a full tablet report...
I20260928 03:03:08.095345 31750 ts_tablet_manager.cc:1428] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 03:03:08.125312 31757 raft_consensus.cc:482] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:03:08.125578 31757 raft_consensus.cc:504] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 46497 } } peers { permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 } } peers { permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 } }
I20260928 03:03:08.127192 31757 leader_election.cc:288] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2271114dfc8d47cb921a2e15583e8c88 (127.12.182.65:44721), 76b698aa24d344d8a9613d8f4a9b3fb8 (127.12.182.66:35101)
I20260928 03:03:08.132582 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.127692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:47247 (local address 127.12.182.66:35101)
0928 03:03:08.128048 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:08.128053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:08.128073 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:03:08.128176 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:08.128182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:08.128244 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:08.128315 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:08.128689 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.129403 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.130992 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.131767 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.131925 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.131998 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:08.132229 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 03:03:08.132348 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260928 03:03:08.132601 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.127623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:35101 (local address 127.12.182.67:47247)
0928 03:03:08.127912 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:08.127933 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:08.128057 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:08.128359 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:08.128364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:08.128390 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:08.128552 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:08.128560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.129542 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.129548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:08.130844 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:08.130855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.131910 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.131916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:08.132085 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.132108 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 03:03:08.132193 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 03:03:08.132315 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 03:03:08.133320 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.127860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:44721 (local address 127.12.182.67:46901)
0928 03:03:08.128534 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:08.128548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:08.128669 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:08.128959 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:08.128963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:08.128986 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:08.129288 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:08.129296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.130306 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.130310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:08.131585 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:08.131596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.132648 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.132654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:08.132828 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.132859 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 03:03:08.132946 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 03:03:08.133058 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":370,"threads_started":1}
I20260928 03:03:08.133315 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.127677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:46901 (local address 127.12.182.65:44721)
0928 03:03:08.127917 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:08.127923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:08.128646 (+ 723us) server_negotiation.cc:407] Connection header received
0928 03:03:08.128799 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:08.128803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:08.128862 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:08.128934 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:08.129445 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.130169 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.131732 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.132463 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.132599 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.132675 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:08.132993 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 03:03:08.133093 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 03:03:08.133702 31552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc887765be49548971f8ef2fd3bf96" candidate_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" is_pre_election: true
I20260928 03:03:08.134197 31552 raft_consensus.cc:2454] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b8cf135c4fa4e7ab8644a2ad63db7d2 in term 0.
I20260928 03:03:08.134321 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc887765be49548971f8ef2fd3bf96" candidate_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2271114dfc8d47cb921a2e15583e8c88" is_pre_election: true
I20260928 03:03:08.134836 31620 leader_election.cc:302] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b698aa24d344d8a9613d8f4a9b3fb8, 9b8cf135c4fa4e7ab8644a2ad63db7d2; no voters:
I20260928 03:03:08.134860 31418 raft_consensus.cc:2454] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b8cf135c4fa4e7ab8644a2ad63db7d2 in term 0.
I20260928 03:03:08.135193 31757 raft_consensus.cc:2781] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:03:08.135352 31757 raft_consensus.cc:482] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:03:08.135463 31757 raft_consensus.cc:3037] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:08.137830 31757 raft_consensus.cc:504] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 46497 } } peers { permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 } } peers { permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 } }
I20260928 03:03:08.138423 31757 leader_election.cc:288] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [CANDIDATE]: Term 1 election: Requested vote from peers 2271114dfc8d47cb921a2e15583e8c88 (127.12.182.65:44721), 76b698aa24d344d8a9613d8f4a9b3fb8 (127.12.182.66:35101)
I20260928 03:03:08.138814 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc887765be49548971f8ef2fd3bf96" candidate_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2271114dfc8d47cb921a2e15583e8c88"
I20260928 03:03:08.138861 31552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc887765be49548971f8ef2fd3bf96" candidate_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8"
I20260928 03:03:08.138988 31418 raft_consensus.cc:3037] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:08.139024 31552 raft_consensus.cc:3037] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:08.141093 31418 raft_consensus.cc:2454] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b8cf135c4fa4e7ab8644a2ad63db7d2 in term 1.
I20260928 03:03:08.141099 31552 raft_consensus.cc:2454] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b8cf135c4fa4e7ab8644a2ad63db7d2 in term 1.
I20260928 03:03:08.141454 31620 leader_election.cc:302] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b698aa24d344d8a9613d8f4a9b3fb8, 9b8cf135c4fa4e7ab8644a2ad63db7d2; no voters:
I20260928 03:03:08.141721 31757 raft_consensus.cc:2781] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:08.142115 31757 raft_consensus.cc:686] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 1 LEADER]: Becoming Leader. State: Replica: 9b8cf135c4fa4e7ab8644a2ad63db7d2, State: Running, Role: LEADER
I20260928 03:03:08.142545 31757 consensus_queue.cc:237] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 46497 } } peers { permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 } } peers { permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 } }
I20260928 03:03:08.146880 31273 catalog_manager.cc:5495] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b8cf135c4fa4e7ab8644a2ad63db7d2 (127.12.182.67). New cstate: current_term: 1 leader_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8cf135c4fa4e7ab8644a2ad63db7d2" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 46497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 } health_report { overall_health: UNKNOWN } } }
W20260928 03:03:08.156010 31464 tablet.cc:2369] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:03:08.171430 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.166581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.12.182.67:46497)
0928 03:03:08.166736 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:08.166741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:08.166758 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:03:08.166875 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:08.166879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:08.166954 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:08.167065 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:08.167355 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.168077 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.169403 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.169851 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.169938 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.170458 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:08.170916 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:08.171045 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:08.171200 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 03:03:08.171237 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 03:03:08.179153 31552 raft_consensus.cc:1261] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8 [term 1 FOLLOWER]: Refusing update from remote peer 9b8cf135c4fa4e7ab8644a2ad63db7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:08.179158 31418 raft_consensus.cc:1261] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88 [term 1 FOLLOWER]: Refusing update from remote peer 9b8cf135c4fa4e7ab8644a2ad63db7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:08.180037 31757 consensus_queue.cc:1035] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2271114dfc8d47cb921a2e15583e8c88" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 44721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:08.180328 31759 consensus_queue.cc:1035] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b698aa24d344d8a9613d8f4a9b3fb8" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:08.203713 31274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:08.204025 31274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:08.207973 31274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8cab689fab58497382fcac54284a8420: Sending DeleteTablet for 3 replicas of tablet 13bc887765be49548971f8ef2fd3bf96
I20260928 03:03:08.209059 31666 tablet_service.cc:1514] Processing DeleteTablet for tablet 13bc887765be49548971f8ef2fd3bf96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:08 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:47456
I20260928 03:03:08.209259 31532 tablet_service.cc:1514] Processing DeleteTablet for tablet 13bc887765be49548971f8ef2fd3bf96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:08 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:42566
I20260928 03:03:08.209823 31398 tablet_service.cc:1514] Processing DeleteTablet for tablet 13bc887765be49548971f8ef2fd3bf96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:08 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:53644
I20260928 03:03:08.209977 31771 tablet_replica.cc:330] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: stopping tablet replica
I20260928 03:03:08.210376 31771 raft_consensus.cc:2229] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:03:08.210983 31771 raft_consensus.cc:2258] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:08.214373 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 31335
I20260928 03:03:08.221877 31772 tablet_replica.cc:330] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: stopping tablet replica
I20260928 03:03:08.222493 31772 raft_consensus.cc:2229] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:08.223140 31772 raft_consensus.cc:2258] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:08.223482 31773 tablet_replica.cc:330] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: stopping tablet replica
I20260928 03:03:08.223899 31773 raft_consensus.cc:2229] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:08.224371 31773 raft_consensus.cc:2258] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:08.225342 31772 ts_tablet_manager.cc:1905] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:08.226275 31773 ts_tablet_manager.cc:1905] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:08.226711 31771 ts_tablet_manager.cc:1905] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:08.232647 31773 ts_tablet_manager.cc:1918] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:08.232657 31772 ts_tablet_manager.cc:1918] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:08.232807 31773 log.cc:1196] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/13bc887765be49548971f8ef2fd3bf96
I20260928 03:03:08.232941 31772 log.cc:1196] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/13bc887765be49548971f8ef2fd3bf96
I20260928 03:03:08.233309 31773 ts_tablet_manager.cc:1939] T 13bc887765be49548971f8ef2fd3bf96 P 2271114dfc8d47cb921a2e15583e8c88: Deleting consensus metadata
I20260928 03:03:08.233547 31772 ts_tablet_manager.cc:1939] T 13bc887765be49548971f8ef2fd3bf96 P 76b698aa24d344d8a9613d8f4a9b3fb8: Deleting consensus metadata
I20260928 03:03:08.235129 31259 catalog_manager.cc:4841] TS 76b698aa24d344d8a9613d8f4a9b3fb8 (127.12.182.66:35101): tablet 13bc887765be49548971f8ef2fd3bf96 (table test-table [id=1a73133a046c42eba80303c41194ca5f]) successfully deleted
I20260928 03:03:08.235438 31261 catalog_manager.cc:4841] TS 2271114dfc8d47cb921a2e15583e8c88 (127.12.182.65:44721): tablet 13bc887765be49548971f8ef2fd3bf96 (table test-table [id=1a73133a046c42eba80303c41194ca5f]) successfully deleted
I20260928 03:03:08.237920 31771 ts_tablet_manager.cc:1918] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:08.238087 31771 log.cc:1196] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/13bc887765be49548971f8ef2fd3bf96
I20260928 03:03:08.238581 31771 ts_tablet_manager.cc:1939] T 13bc887765be49548971f8ef2fd3bf96 P 9b8cf135c4fa4e7ab8644a2ad63db7d2: Deleting consensus metadata
I20260928 03:03:08.240331 31260 catalog_manager.cc:4841] TS 9b8cf135c4fa4e7ab8644a2ad63db7d2 (127.12.182.67:46497): tablet 13bc887765be49548971f8ef2fd3bf96 (table test-table [id=1a73133a046c42eba80303c41194ca5f]) successfully deleted
Sep 28 03:03:08 dist-test-slave-k5rg krb5kdc[31229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 03:03:08.256999 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.216828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.12.182.65:44721)
0928 03:03:08.217010 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:08.217016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:08.231160 (+ 14144us) server_negotiation.cc:407] Connection header received
0928 03:03:08.231644 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:08.231650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:08.231746 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:08.231820 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:08.237756 (+ 5936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.238563 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.239960 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.240505 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.240619 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.251458 (+ 10839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:08.251472 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:08.251487 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:08.251526 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:08.254297 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:08.254959 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:08.254966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:08.254968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:08.255091 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:08.255431 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:08.255435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:08.255437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:08.255824 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:08.256103 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:08.256424 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 03:03:08.256662 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 03:03:08.257699 31458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:08.374739 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 31335
I20260928 03:03:08.397164 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 31469
Sep 28 03:03:08 dist-test-slave-k5rg krb5kdc[31229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 03:03:08.412360 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.398019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.12.182.66:35101)
0928 03:03:08.398374 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:08.398378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:08.398392 (+ 14us) server_negotiation.cc:407] Connection header received
0928 03:03:08.398451 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:08.398455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:08.398514 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:08.398599 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:08.399833 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.400547 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.401959 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.402337 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.402444 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.406729 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:08.406746 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:08.406762 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:08.406802 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:08.409664 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:08.410357 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:08.410364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:08.410366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:08.410494 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:08.410839 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:08.410842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:08.410844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:08.411227 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:08.411492 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:08.411841 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 03:03:08.412056 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260928 03:03:08.413014 31592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:08.510553 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 31469
I20260928 03:03:08.529520 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 31603
Sep 28 03:03:08 dist-test-slave-k5rg krb5kdc[31229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 03:03:08.545328 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:08.530351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.12.182.67:46497)
0928 03:03:08.530530 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:08.530536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:08.530555 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:03:08.530690 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:08.530694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:08.530763 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:08.530853 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:08.532068 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.532893 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.534283 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:08.534727 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:08.534833 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:08.539454 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:08.539465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:08.539484 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:08.539517 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:08.542600 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:08.543247 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:08.543254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:08.543260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:08.543413 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:08.543781 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:08.543785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:08.543787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:08.544159 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:08.544462 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:08.544741 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 03:03:08.544952 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:03:08.546103 31726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:08.667281 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 31603
I20260928 03:03:08.682843 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 31242
I20260928 03:03:08.683938 31304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:08.794430 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 31242
2026-09-28T03:03:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4141 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T03:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:08Z Disabled control of system clock
I20260928 03:03:08.862906 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33117
--webserver_interface=127.12.182.126
--webserver_port=0
--builtin_ntp_servers=127.12.182.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:03:09.176407 31782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:09.176690 31782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:09.176779 31782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:09.184032 31782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:09.184149 31782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:09.184185 31782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:09.184239 31782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:09.184283 31782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:09.193389 31782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:09.194658 31782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:09.196346 31782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:09.207290 31790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:09.207253 31787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:09.207329 31788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:09.208493 31782 server_base.cc:1034] running on GCE node
I20260928 03:03:09.209276 31782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:09.210676 31782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:09.211882 31782 hybrid_clock.cc:648] HybridClock initialized: now 1790564589211821 us; error 50 us; skew 500 ppm
I20260928 03:03:09.212200 31782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:09.214318 31782 webserver.cc:460] Webserver started at http://127.12.182.126:34509/ using document root <none> and password file <none>
I20260928 03:03:09.214952 31782 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:09.215049 31782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:09.215339 31782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:09.218094 31782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "608a15b5cc5443e1a968dc167c698fae"
format_stamp: "Formatted at 2026-09-28 03:03:09 on dist-test-slave-k5rg"
server_key: "cf3a52f1be7314a9cb47ab3b1fdc6c6f"
server_key_iv: "631c84d0d6a74bebf973d55e3330c1ad"
server_key_version: "encryptionkey@0"
I20260928 03:03:09.218850 31782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "608a15b5cc5443e1a968dc167c698fae"
format_stamp: "Formatted at 2026-09-28 03:03:09 on dist-test-slave-k5rg"
server_key: "cf3a52f1be7314a9cb47ab3b1fdc6c6f"
server_key_iv: "631c84d0d6a74bebf973d55e3330c1ad"
server_key_version: "encryptionkey@0"
I20260928 03:03:09.223651 31782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 03:03:09.226845 31796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:09.228042 31782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260928 03:03:09.228197 31782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "608a15b5cc5443e1a968dc167c698fae"
format_stamp: "Formatted at 2026-09-28 03:03:09 on dist-test-slave-k5rg"
server_key: "cf3a52f1be7314a9cb47ab3b1fdc6c6f"
server_key_iv: "631c84d0d6a74bebf973d55e3330c1ad"
server_key_version: "encryptionkey@0"
I20260928 03:03:09.228389 31782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:09.250069 31782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:09.250885 31782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:09.251149 31782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:09.267709 31782 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:33117
I20260928 03:03:09.267728 31847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:33117 every 8 connection(s)
I20260928 03:03:09.268980 31782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:09.272557 31848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:09.273031 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 31782
I20260928 03:03:09.273190 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:09.273653 13017 external_mini_cluster.cc:1428] Setting key e51078db94593e83e16d811135f64645
I20260928 03:03:09.285277 31848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae: Bootstrap starting.
I20260928 03:03:09.286975 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:09.275479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.12.182.126:33117)
0928 03:03:09.276432 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:09.276442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:09.276484 (+ 42us) server_negotiation.cc:407] Connection header received
0928 03:03:09.278218 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:09.278239 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:09.278666 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:09.278986 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:09.279570 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:09.280836 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:09.282728 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:09.283411 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:09.283547 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:09.284741 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:09.284749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:09.284779 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:09.285293 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:09.285368 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:09.285389 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:09.285490 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:09.285734 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 03:03:09.286069 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":349,"threads_started":1}
I20260928 03:03:09.288718 31848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:09.289846 31848 log.cc:824] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:09.292366 31848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae: No bootstrap required, opened a new log
I20260928 03:03:09.299505 31848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608a15b5cc5443e1a968dc167c698fae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33117 } }
I20260928 03:03:09.299860 31848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:09.299947 31848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608a15b5cc5443e1a968dc167c698fae, State: Initialized, Role: FOLLOWER
I20260928 03:03:09.300484 31848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608a15b5cc5443e1a968dc167c698fae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33117 } }
I20260928 03:03:09.300748 31848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:09.300892 31848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:09.301020 31848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:09.303102 31848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608a15b5cc5443e1a968dc167c698fae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33117 } }
I20260928 03:03:09.303524 31848 leader_election.cc:302] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 608a15b5cc5443e1a968dc167c698fae; no voters:
I20260928 03:03:09.304054 31848 leader_election.cc:288] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:09.304200 31853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:09.305049 31853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [term 1 LEADER]: Becoming Leader. State: Replica: 608a15b5cc5443e1a968dc167c698fae, State: Running, Role: LEADER
I20260928 03:03:09.305585 31853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608a15b5cc5443e1a968dc167c698fae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33117 } }
I20260928 03:03:09.306392 31848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:09.308441 31855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 608a15b5cc5443e1a968dc167c698fae. Latest consensus state: current_term: 1 leader_uuid: "608a15b5cc5443e1a968dc167c698fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608a15b5cc5443e1a968dc167c698fae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33117 } } }
I20260928 03:03:09.308598 31854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "608a15b5cc5443e1a968dc167c698fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608a15b5cc5443e1a968dc167c698fae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33117 } } }
I20260928 03:03:09.308866 31855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:09.308893 31854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:09.309821 31861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:09.315778 31861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:09.324542 31861 catalog_manager.cc:1347] Generated new cluster ID: 8459bb72e5054b239bd4c7dc7f57290a
I20260928 03:03:09.324674 31861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:09.348160 31861 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:09.349932 31861 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:09.360023 31861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae: Generated new TSK 0
I20260928 03:03:09.361038 31861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:09.371078 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33117
--builtin_ntp_servers=127.12.182.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:03:09.689966 31872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:09.690238 31872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:09.690367 31872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:09.697321 31872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:09.697419 31872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:09.697564 31872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:03:09.706341 31872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:09.707598 31872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:09.709275 31872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:09.721338 31877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:09.721293 31878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:09.721370 31880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:09.722390 31872 server_base.cc:1034] running on GCE node
I20260928 03:03:09.722929 31872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:09.723814 31872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:09.725008 31872 hybrid_clock.cc:648] HybridClock initialized: now 1790564589724951 us; error 54 us; skew 500 ppm
I20260928 03:03:09.725297 31872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:09.727363 31872 webserver.cc:460] Webserver started at http://127.12.182.65:36381/ using document root <none> and password file <none>
I20260928 03:03:09.727974 31872 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:09.728068 31872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:09.728353 31872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:09.731101 31872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "d9fb8fcc98a04c09a43777d35e6486cd"
format_stamp: "Formatted at 2026-09-28 03:03:09 on dist-test-slave-k5rg"
server_key: "216a1776a21e415ad41064b17167808b"
server_key_iv: "0300ead4d68930861aee5cc66e2d055b"
server_key_version: "encryptionkey@0"
I20260928 03:03:09.731913 31872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "d9fb8fcc98a04c09a43777d35e6486cd"
format_stamp: "Formatted at 2026-09-28 03:03:09 on dist-test-slave-k5rg"
server_key: "216a1776a21e415ad41064b17167808b"
server_key_iv: "0300ead4d68930861aee5cc66e2d055b"
server_key_version: "encryptionkey@0"
I20260928 03:03:09.736747 31872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 03:03:09.740031 31886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:09.741297 31872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:03:09.741443 31872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "d9fb8fcc98a04c09a43777d35e6486cd"
format_stamp: "Formatted at 2026-09-28 03:03:09 on dist-test-slave-k5rg"
server_key: "216a1776a21e415ad41064b17167808b"
server_key_iv: "0300ead4d68930861aee5cc66e2d055b"
server_key_version: "encryptionkey@0"
I20260928 03:03:09.741657 31872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:09.759286 31872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:09.760035 31872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:09.760314 31872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:09.761193 31872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:09.762636 31872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:09.762733 31872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:09.762837 31872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:09.762907 31872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:09.794596 31872 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:43993
I20260928 03:03:09.794643 31998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:43993 every 8 connection(s)
I20260928 03:03:09.796188 31872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:03:09.800689 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 31872
I20260928 03:03:09.800891 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 03:03:09.801452 13017 external_mini_cluster.cc:1428] Setting key 0b403d5c88346b70fe3a4e9b5b4daaa1
I20260928 03:03:09.805084 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33117
--builtin_ntp_servers=127.12.182.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:03:09.810786 32001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:09.798899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:53585 (local address 127.12.182.126:33117)
0928 03:03:09.799445 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:09.799452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:09.803363 (+ 3911us) server_negotiation.cc:407] Connection header received
0928 03:03:09.803418 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:09.803421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:09.803481 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:09.803561 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:09.804947 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:09.805879 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:09.807809 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:09.808216 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:09.808337 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:09.809935 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:09.809942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:09.809945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:09.809965 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:09.810012 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:09.810024 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:09.810105 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:09.810380 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 03:03:09.810478 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":514,"threads_started":1}
I20260928 03:03:09.812407 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:09.799613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33117 (local address 127.12.182.65:53585)
0928 03:03:09.800726 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:09.800777 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:09.803015 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:09.803868 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:09.803880 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:09.803999 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:09.804721 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:09.804758 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:09.806031 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:09.806037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:09.807659 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:09.807677 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:09.808431 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:09.808439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:09.808732 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:09.809477 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:03:09.809481 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:09.809541 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:03:09.809741 (+ 200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:09.810118 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:09.810129 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:09.810138 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 03:03:09.810336 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 03:03:09.810819 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":307,"threads_started":1}
I20260928 03:03:09.813997 31999 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33117
I20260928 03:03:09.814280 31999 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:09.815050 31999 heartbeater.cc:511] Master 127.12.182.126:33117 requested a full tablet report, sending...
I20260928 03:03:09.817159 31813 ts_manager.cc:194] Registered new tserver with Master: d9fb8fcc98a04c09a43777d35e6486cd (127.12.182.65:43993)
I20260928 03:03:09.818631 31813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.65:53585
W20260928 03:03:10.119441 32003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:10.119696 32003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:10.119825 32003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:10.126968 32003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:10.127063 32003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:10.127195 32003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:03:10.136291 32003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:10.137521 32003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:10.139197 32003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:10.151363 32009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:10.151315 32008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:10.151551 32011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:10.152357 32003 server_base.cc:1034] running on GCE node
I20260928 03:03:10.152927 32003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:10.153869 32003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:10.155067 32003 hybrid_clock.cc:648] HybridClock initialized: now 1790564590155014 us; error 50 us; skew 500 ppm
I20260928 03:03:10.155350 32003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:10.157485 32003 webserver.cc:460] Webserver started at http://127.12.182.66:43427/ using document root <none> and password file <none>
I20260928 03:03:10.158177 32003 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:10.158281 32003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:10.158560 32003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:10.161254 32003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "4b5174ef4a7142018cc0a1220254606b"
format_stamp: "Formatted at 2026-09-28 03:03:10 on dist-test-slave-k5rg"
server_key: "884d1d271485b258b84d4df36f6056e7"
server_key_iv: "083e1ad07cb220d7f0d6b862dd8b262d"
server_key_version: "encryptionkey@0"
I20260928 03:03:10.162004 32003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "4b5174ef4a7142018cc0a1220254606b"
format_stamp: "Formatted at 2026-09-28 03:03:10 on dist-test-slave-k5rg"
server_key: "884d1d271485b258b84d4df36f6056e7"
server_key_iv: "083e1ad07cb220d7f0d6b862dd8b262d"
server_key_version: "encryptionkey@0"
I20260928 03:03:10.166904 32003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 03:03:10.170145 32017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:10.171370 32003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:03:10.171514 32003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "4b5174ef4a7142018cc0a1220254606b"
format_stamp: "Formatted at 2026-09-28 03:03:10 on dist-test-slave-k5rg"
server_key: "884d1d271485b258b84d4df36f6056e7"
server_key_iv: "083e1ad07cb220d7f0d6b862dd8b262d"
server_key_version: "encryptionkey@0"
I20260928 03:03:10.171718 32003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:10.188076 32003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:10.188866 32003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:10.189141 32003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:10.190114 32003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:10.191533 32003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:10.191628 32003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:10.191736 32003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:10.191802 32003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:10.221041 32003 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:45023
I20260928 03:03:10.221112 32129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:45023 every 8 connection(s)
I20260928 03:03:10.222280 32003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:03:10.231043 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 32003
I20260928 03:03:10.231244 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 03:03:10.231704 13017 external_mini_cluster.cc:1428] Setting key a267370d3eaf9872926767d9454a7ccd
I20260928 03:03:10.233940 32001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.224657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:40827 (local address 127.12.182.126:33117)
0928 03:03:10.224841 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:10.224852 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:10.226102 (+ 1250us) server_negotiation.cc:407] Connection header received
0928 03:03:10.227801 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:10.227805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:10.227870 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:10.227950 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:10.228913 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.229579 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.231253 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.231775 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.231882 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.233194 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:10.233199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:10.233204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:10.233224 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:10.233261 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:10.233275 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:10.233356 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:10.233578 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 03:03:10.233713 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:03:10.234645 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33117
--builtin_ntp_servers=127.12.182.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:03:10.235170 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.225117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33117 (local address 127.12.182.66:40827)
0928 03:03:10.225978 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:10.226016 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:10.227573 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:10.228124 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:10.228131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:10.228217 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:10.228785 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.228803 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.229739 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.229743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.231115 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.231130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.231983 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.231991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.232264 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.232904 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:03:10.232907 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:10.232943 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:03:10.233012 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:10.233371 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:10.233382 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:10.233389 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 03:03:10.233548 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 03:03:10.233942 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":252,"threads_started":1}
I20260928 03:03:10.236414 32130 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33117
I20260928 03:03:10.236739 32130 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:10.237540 32130 heartbeater.cc:511] Master 127.12.182.126:33117 requested a full tablet report, sending...
I20260928 03:03:10.238929 31813 ts_manager.cc:194] Registered new tserver with Master: 4b5174ef4a7142018cc0a1220254606b (127.12.182.66:45023)
I20260928 03:03:10.239675 31813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.66:40827
W20260928 03:03:10.545758 32133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:10.546005 32133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:10.546113 32133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:10.553175 32133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:10.553268 32133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:10.553400 32133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:03:10.562240 32133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:33117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:10.563479 32133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:10.565155 32133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:10.577316 32141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:10.577487 32138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:10.577503 32139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:10.578652 32133 server_base.cc:1034] running on GCE node
I20260928 03:03:10.579273 32133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:10.580188 32133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:10.581391 32133 hybrid_clock.cc:648] HybridClock initialized: now 1790564590581347 us; error 31 us; skew 500 ppm
I20260928 03:03:10.581709 32133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:10.583837 32133 webserver.cc:460] Webserver started at http://127.12.182.67:41997/ using document root <none> and password file <none>
I20260928 03:03:10.584468 32133 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:10.584565 32133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:10.584857 32133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:10.587579 32133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "9911aa7496d04ab382aab49bed9f822d"
format_stamp: "Formatted at 2026-09-28 03:03:10 on dist-test-slave-k5rg"
server_key: "5e868ce8bd1cce96ee0ee8321a4c3cc1"
server_key_iv: "a3bd40d25c8aad856ae259f46d46958b"
server_key_version: "encryptionkey@0"
I20260928 03:03:10.588326 32133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "9911aa7496d04ab382aab49bed9f822d"
format_stamp: "Formatted at 2026-09-28 03:03:10 on dist-test-slave-k5rg"
server_key: "5e868ce8bd1cce96ee0ee8321a4c3cc1"
server_key_iv: "a3bd40d25c8aad856ae259f46d46958b"
server_key_version: "encryptionkey@0"
I20260928 03:03:10.593169 32133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 03:03:10.596398 32147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:10.597584 32133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 03:03:10.597770 32133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "9911aa7496d04ab382aab49bed9f822d"
format_stamp: "Formatted at 2026-09-28 03:03:10 on dist-test-slave-k5rg"
server_key: "5e868ce8bd1cce96ee0ee8321a4c3cc1"
server_key_iv: "a3bd40d25c8aad856ae259f46d46958b"
server_key_version: "encryptionkey@0"
I20260928 03:03:10.597954 32133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:10.615895 32133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:10.616652 32133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:10.616921 32133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:10.617882 32133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:10.619266 32133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:10.619364 32133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:10.619467 32133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:10.619536 32133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:10.648876 32133 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:44723
I20260928 03:03:10.648926 32259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:44723 every 8 connection(s)
I20260928 03:03:10.650156 32133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:03:10.658504 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 32133
I20260928 03:03:10.658752 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 03:03:10.659309 13017 external_mini_cluster.cc:1428] Setting key 74aca6c29736e4bcc424c218306616eb
I20260928 03:03:10.662479 32001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.652468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:44747 (local address 127.12.182.126:33117)
0928 03:03:10.652660 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:10.652665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:10.654028 (+ 1363us) server_negotiation.cc:407] Connection header received
0928 03:03:10.655688 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:10.655693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:10.655764 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:10.655849 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:10.656804 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.657512 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.659633 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.660074 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.660199 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.661664 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:10.661669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:10.661672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:10.661689 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:10.661720 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:10.661735 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:10.661804 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:10.662044 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 03:03:10.662142 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:03:10.663465 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.652952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33117 (local address 127.12.182.67:44747)
0928 03:03:10.653882 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:10.653917 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:10.655460 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:10.656031 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:10.656038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:10.656137 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:10.656665 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.656682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.657700 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.657704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.659492 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.659507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.660304 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.660310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.660581 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.661284 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:03:10.661288 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:10.661349 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:03:10.661457 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:10.661831 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:10.661841 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:10.661850 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 03:03:10.662019 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 03:03:10.662382 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":273,"threads_started":1}
I20260928 03:03:10.664706 32260 heartbeater.cc:347] Connected to a master server at 127.12.182.126:33117
I20260928 03:03:10.664959 32260 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:10.665853 32260 heartbeater.cc:511] Master 127.12.182.126:33117 requested a full tablet report, sending...
I20260928 03:03:10.667472 31813 ts_manager.cc:194] Registered new tserver with Master: 9911aa7496d04ab382aab49bed9f822d (127.12.182.67:44723)
I20260928 03:03:10.668256 31813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.67:44747
I20260928 03:03:10.674300 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:03:10.683820 32001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.678313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.12.182.126:33117)
0928 03:03:10.678476 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:10.678480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:10.678869 (+ 389us) server_negotiation.cc:407] Connection header received
0928 03:03:10.679028 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:10.679032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:10.679107 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:10.679198 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:10.679634 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.680323 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.681691 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.682092 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.682194 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.683188 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:10.683192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:10.683195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:10.683212 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:10.683274 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:10.683287 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:10.683393 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:10.683520 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 03:03:10.683567 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 03:03:10.687588 31813 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:03:10.694157 31813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:03:10.724740 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.705758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.12.182.65:43993)
0928 03:03:10.716741 (+ 10983us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:10.716749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:10.716778 (+ 29us) server_negotiation.cc:407] Connection header received
0928 03:03:10.716865 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:10.716871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:10.717069 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:10.717238 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:10.718566 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.719933 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.722453 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.723681 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.723801 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.724050 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:10.724258 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 03:03:10.724419 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10826,"thread_start_us":10726,"threads_started":1}
I20260928 03:03:10.724761 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.705651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43993 (local address 127.0.0.1:35366)
0928 03:03:10.715558 (+ 9907us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:10.715592 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:10.715773 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:10.717465 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:10.717469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:10.717492 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:10.718379 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.718397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.720059 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.720064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.722223 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.722239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.723844 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.723850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.724036 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.724086 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 03:03:10.724185 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 03:03:10.724390 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9658,"mutex_wait_us":38,"thread_start_us":9544,"threads_started":1}
I20260928 03:03:10.725370 32275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.706057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.12.182.67:44723)
0928 03:03:10.715170 (+ 9113us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:10.715179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:10.716268 (+ 1089us) server_negotiation.cc:407] Connection header received
0928 03:03:10.716433 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:10.716441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:10.716712 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:10.716903 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:10.718918 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.720167 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.722457 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.723762 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.723886 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.724132 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:10.724859 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 03:03:10.725026 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8944,"thread_start_us":8791,"threads_started":1}
I20260928 03:03:10.725888 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.705743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:44723 (local address 127.0.0.1:51956)
0928 03:03:10.716153 (+ 10410us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:10.716167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:10.716278 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:10.717209 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:10.717217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:10.717247 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:10.718747 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.718762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.720305 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.720311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.722207 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.722222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.724259 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.724265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.724427 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.724457 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 03:03:10.724531 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 03:03:10.724646 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10315,"mutex_wait_us":578,"thread_start_us":10199,"threads_started":1}
I20260928 03:03:10.726671 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.717262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:45023 (local address 127.0.0.1:60442)
0928 03:03:10.718524 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:10.718539 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:10.718634 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:10.719140 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:10.719144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:10.719168 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:10.719506 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.719517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.721101 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.721107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.723168 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.723183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.726184 (+ 3001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.726191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.726307 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.726328 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 03:03:10.726385 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 03:03:10.726461 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"mutex_wait_us":2,"thread_start_us":999,"threads_started":1}
I20260928 03:03:10.728739 31934 tablet_service.cc:1467] Processing CreateTablet for tablet 73c437b1fa1948d1ae286bf24d11495d (DEFAULT_TABLE table=test-table [id=6e7fc8469de34537a302fdb50b68ff3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:10.728848 32195 tablet_service.cc:1467] Processing CreateTablet for tablet 73c437b1fa1948d1ae286bf24d11495d (DEFAULT_TABLE table=test-table [id=6e7fc8469de34537a302fdb50b68ff3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:10.730032 31934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c437b1fa1948d1ae286bf24d11495d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:10.730159 32195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c437b1fa1948d1ae286bf24d11495d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:10.730319 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.716616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.12.182.66:45023)
0928 03:03:10.717097 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:10.717104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:10.718640 (+ 1536us) server_negotiation.cc:407] Connection header received
0928 03:03:10.718750 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:10.718757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:10.718960 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:10.719140 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:10.719668 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.720961 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.723372 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.725829 (+ 2457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.725956 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.729760 (+ 3804us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:10.729836 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 03:03:10.729993 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":236,"threads_started":1}
I20260928 03:03:10.733680 32065 tablet_service.cc:1467] Processing CreateTablet for tablet 73c437b1fa1948d1ae286bf24d11495d (DEFAULT_TABLE table=test-table [id=6e7fc8469de34537a302fdb50b68ff3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:10.735033 32065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c437b1fa1948d1ae286bf24d11495d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:10.740881 32278 tablet_bootstrap.cc:492] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: Bootstrap starting.
I20260928 03:03:10.743721 32279 tablet_bootstrap.cc:492] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd: Bootstrap starting.
I20260928 03:03:10.744624 32278 tablet_bootstrap.cc:654] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:10.746002 32278 log.cc:824] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:10.747223 32279 tablet_bootstrap.cc:654] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:10.748172 32281 tablet_bootstrap.cc:492] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: Bootstrap starting.
I20260928 03:03:10.748333 32279 log.cc:824] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:10.749672 32278 tablet_bootstrap.cc:492] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: No bootstrap required, opened a new log
I20260928 03:03:10.749946 32278 ts_tablet_manager.cc:1397] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260928 03:03:10.750910 32281 tablet_bootstrap.cc:654] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:10.751266 32279 tablet_bootstrap.cc:492] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd: No bootstrap required, opened a new log
I20260928 03:03:10.751524 32279 ts_tablet_manager.cc:1397] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 03:03:10.751843 32281 log.cc:824] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:10.754165 32281 tablet_bootstrap.cc:492] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: No bootstrap required, opened a new log
I20260928 03:03:10.754408 32281 ts_tablet_manager.cc:1397] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 03:03:10.760059 32278 raft_consensus.cc:348] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 } } peers { permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 } } peers { permanent_uuid: "4b5174ef4a7142018cc0a1220254606b" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 45023 } }
I20260928 03:03:10.760557 32278 raft_consensus.cc:374] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:10.760658 32278 raft_consensus.cc:729] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9911aa7496d04ab382aab49bed9f822d, State: Initialized, Role: FOLLOWER
I20260928 03:03:10.761307 32278 consensus_queue.cc:260] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 } } peers { permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 } } peers { permanent_uuid: "4b5174ef4a7142018cc0a1220254606b" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 45023 } }
I20260928 03:03:10.762048 32281 raft_consensus.cc:348] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 } } peers { permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 } } peers { permanent_uuid: "4b5174ef4a7142018cc0a1220254606b" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 45023 } }
I20260928 03:03:10.762447 32260 heartbeater.cc:503] Master 127.12.182.126:33117 was elected leader, sending a full tablet report...
I20260928 03:03:10.762454 32281 raft_consensus.cc:374] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:10.762560 32281 raft_consensus.cc:729] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5174ef4a7142018cc0a1220254606b, State: Initialized, Role: FOLLOWER
I20260928 03:03:10.763057 32278 ts_tablet_manager.cc:1428] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260928 03:03:10.763058 32281 consensus_queue.cc:260] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 } } peers { permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 } } peers { permanent_uuid: "4b5174ef4a7142018cc0a1220254606b" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 45023 } }
I20260928 03:03:10.762022 32279 raft_consensus.cc:348] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 } } peers { permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 } } peers { permanent_uuid: "4b5174ef4a7142018cc0a1220254606b" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 45023 } }
I20260928 03:03:10.763552 32279 raft_consensus.cc:374] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:10.763670 32279 raft_consensus.cc:729] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fb8fcc98a04c09a43777d35e6486cd, State: Initialized, Role: FOLLOWER
I20260928 03:03:10.764201 32130 heartbeater.cc:503] Master 127.12.182.126:33117 was elected leader, sending a full tablet report...
I20260928 03:03:10.764788 32281 ts_tablet_manager.cc:1428] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260928 03:03:10.765244 32279 consensus_queue.cc:260] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 } } peers { permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 } } peers { permanent_uuid: "4b5174ef4a7142018cc0a1220254606b" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 45023 } }
I20260928 03:03:10.766381 31999 heartbeater.cc:503] Master 127.12.182.126:33117 was elected leader, sending a full tablet report...
I20260928 03:03:10.767004 32279 ts_tablet_manager.cc:1428] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260928 03:03:10.798341 32000 tablet.cc:2369] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:03:10.901678 32261 tablet.cc:2369] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:03:10.924865 32285 raft_consensus.cc:482] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:03:10.925159 32285 raft_consensus.cc:504] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 } } peers { permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 } } peers { permanent_uuid: "4b5174ef4a7142018cc0a1220254606b" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 45023 } }
I20260928 03:03:10.926772 32285 leader_election.cc:288] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9fb8fcc98a04c09a43777d35e6486cd (127.12.182.65:43993), 9911aa7496d04ab382aab49bed9f822d (127.12.182.67:44723)
I20260928 03:03:10.936076 32287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.927121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:44723 (local address 127.12.182.66:43243)
0928 03:03:10.928044 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:10.928058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:10.928199 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:10.928533 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:10.928539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:10.928578 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:10.929159 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.929169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.930585 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.930590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.933036 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.933045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.935058 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.935065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.935343 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.935389 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 03:03:10.935484 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 03:03:10.935642 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":150,"spinlock_wait_cycles":50816,"thread_start_us":388,"threads_started":1}
I20260928 03:03:10.936084 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.927046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43993 (local address 127.12.182.66:44443)
0928 03:03:10.927973 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:10.927994 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:10.928157 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:10.928534 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:10.928539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:10.928566 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:10.929128 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.929145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.930587 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.930591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.933025 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:10.933041 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.935025 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.935033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:10.935319 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.935393 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 03:03:10.935484 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 03:03:10.935642 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":301,"spinlock_wait_cycles":5376,"thread_start_us":580,"threads_started":1}
I20260928 03:03:10.936440 32275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.927182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:43243 (local address 127.12.182.67:44723)
0928 03:03:10.927397 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:10.927402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:10.928144 (+ 742us) server_negotiation.cc:407] Connection header received
0928 03:03:10.928325 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:10.928330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:10.928410 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:10.928505 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:10.929329 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.930447 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.933411 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.934650 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.934812 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.934883 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:10.935574 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 03:03:10.935963 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:03:10.936807 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:10.927295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:44443 (local address 127.12.182.65:43993)
0928 03:03:10.927495 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:10.927502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:10.928086 (+ 584us) server_negotiation.cc:407] Connection header received
0928 03:03:10.928325 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:10.928330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:10.928413 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:10.928505 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:10.929314 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.930425 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.933386 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:10.934658 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:10.934815 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:10.934883 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:10.935705 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 03:03:10.936201 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 03:03:10.937422 32215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c437b1fa1948d1ae286bf24d11495d" candidate_uuid: "4b5174ef4a7142018cc0a1220254606b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9911aa7496d04ab382aab49bed9f822d" is_pre_election: true
I20260928 03:03:10.937637 31954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c437b1fa1948d1ae286bf24d11495d" candidate_uuid: "4b5174ef4a7142018cc0a1220254606b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" is_pre_election: true
I20260928 03:03:10.937940 32215 raft_consensus.cc:2454] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5174ef4a7142018cc0a1220254606b in term 0.
I20260928 03:03:10.938112 31954 raft_consensus.cc:2454] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5174ef4a7142018cc0a1220254606b in term 0.
I20260928 03:03:10.938585 32018 leader_election.cc:302] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5174ef4a7142018cc0a1220254606b, 9911aa7496d04ab382aab49bed9f822d; no voters:
I20260928 03:03:10.938949 32285 raft_consensus.cc:2781] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:03:10.939083 32285 raft_consensus.cc:482] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:03:10.939172 32285 raft_consensus.cc:3037] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:10.941429 32285 raft_consensus.cc:504] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 } } peers { permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 } } peers { permanent_uuid: "4b5174ef4a7142018cc0a1220254606b" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 45023 } }
I20260928 03:03:10.942049 32285 leader_election.cc:288] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [CANDIDATE]: Term 1 election: Requested vote from peers d9fb8fcc98a04c09a43777d35e6486cd (127.12.182.65:43993), 9911aa7496d04ab382aab49bed9f822d (127.12.182.67:44723)
I20260928 03:03:10.942476 31954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c437b1fa1948d1ae286bf24d11495d" candidate_uuid: "4b5174ef4a7142018cc0a1220254606b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fb8fcc98a04c09a43777d35e6486cd"
I20260928 03:03:10.942669 31954 raft_consensus.cc:3037] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:10.944842 31954 raft_consensus.cc:2454] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5174ef4a7142018cc0a1220254606b in term 1.
I20260928 03:03:10.945214 32020 leader_election.cc:302] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5174ef4a7142018cc0a1220254606b, d9fb8fcc98a04c09a43777d35e6486cd; no voters:
I20260928 03:03:10.945214 32215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c437b1fa1948d1ae286bf24d11495d" candidate_uuid: "4b5174ef4a7142018cc0a1220254606b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9911aa7496d04ab382aab49bed9f822d"
I20260928 03:03:10.945416 32215 raft_consensus.cc:3037] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:10.945549 32285 raft_consensus.cc:2781] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:10.945998 32285 raft_consensus.cc:686] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 1 LEADER]: Becoming Leader. State: Replica: 4b5174ef4a7142018cc0a1220254606b, State: Running, Role: LEADER
I20260928 03:03:10.946544 32285 consensus_queue.cc:237] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 } } peers { permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 } } peers { permanent_uuid: "4b5174ef4a7142018cc0a1220254606b" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 45023 } }
I20260928 03:03:10.947894 32215 raft_consensus.cc:2454] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5174ef4a7142018cc0a1220254606b in term 1.
I20260928 03:03:10.951398 31811 catalog_manager.cc:5495] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b5174ef4a7142018cc0a1220254606b (127.12.182.66). New cstate: current_term: 1 leader_uuid: "4b5174ef4a7142018cc0a1220254606b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b5174ef4a7142018cc0a1220254606b" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 45023 } health_report { overall_health: HEALTHY } } }
W20260928 03:03:10.975055 32131 tablet.cc:2369] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:03:11.011843 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.006757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.12.182.66:45023)
0928 03:03:11.006963 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:11.006970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:11.007053 (+ 83us) server_negotiation.cc:407] Connection header received
0928 03:03:11.007216 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:11.007219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:11.007313 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:11.007427 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:11.007800 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:11.008499 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:11.009889 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:11.010306 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:11.010389 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:11.010855 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:11.011350 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:11.011473 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:11.011548 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 03:03:11.011585 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 03:03:11.019654 32215 raft_consensus.cc:1261] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d [term 1 FOLLOWER]: Refusing update from remote peer 4b5174ef4a7142018cc0a1220254606b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:11.019654 31954 raft_consensus.cc:1261] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd [term 1 FOLLOWER]: Refusing update from remote peer 4b5174ef4a7142018cc0a1220254606b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:11.020655 32289 consensus_queue.cc:1035] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9911aa7496d04ab382aab49bed9f822d" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:11.020978 32285 consensus_queue.cc:1035] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fb8fcc98a04c09a43777d35e6486cd" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 43993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:11.046903 31810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:11.047168 31810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:11.050822 31810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 608a15b5cc5443e1a968dc167c698fae: Sending DeleteTablet for 3 replicas of tablet 73c437b1fa1948d1ae286bf24d11495d
I20260928 03:03:11.051913 31934 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c437b1fa1948d1ae286bf24d11495d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:11 UTC) from {username='slave'} at 127.0.0.1:35366
I20260928 03:03:11.052062 32065 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c437b1fa1948d1ae286bf24d11495d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:11 UTC) from {username='slave'} at 127.0.0.1:60442
I20260928 03:03:11.052043 32195 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c437b1fa1948d1ae286bf24d11495d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:11 UTC) from {username='slave'} at 127.0.0.1:51956
I20260928 03:03:11.053519 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 31872
I20260928 03:03:11.061335 32305 tablet_replica.cc:330] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: stopping tablet replica
I20260928 03:03:11.063484 32305 raft_consensus.cc:2229] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:11.064409 32305 raft_consensus.cc:2258] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:11.066390 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.054489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.12.182.65:43993)
0928 03:03:11.054657 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:11.054673 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:11.056686 (+ 2013us) server_negotiation.cc:407] Connection header received
0928 03:03:11.056946 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:11.056953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:11.057070 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:11.057145 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:11.057838 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:11.059091 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:11.061529 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:11.062194 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:11.062305 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:11.064677 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:11.064688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:11.064703 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:11.065457 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:11.065522 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:11.065547 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:11.065663 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:11.065897 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 03:03:11.066008 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 03:03:11.067466 32303 tablet_replica.cc:330] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd: stopping tablet replica
I20260928 03:03:11.067740 31994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:11.070777 32304 tablet_replica.cc:330] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: stopping tablet replica
I20260928 03:03:11.071097 32304 raft_consensus.cc:2229] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:03:11.071640 32304 raft_consensus.cc:2258] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:11.073745 32304 ts_tablet_manager.cc:1905] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:11.075662 32305 ts_tablet_manager.cc:1905] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:11.081194 32304 ts_tablet_manager.cc:1918] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:11.081439 32304 log.cc:1196] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/73c437b1fa1948d1ae286bf24d11495d
I20260928 03:03:11.082123 32304 ts_tablet_manager.cc:1939] T 73c437b1fa1948d1ae286bf24d11495d P 4b5174ef4a7142018cc0a1220254606b: Deleting consensus metadata
I20260928 03:03:11.082715 32305 ts_tablet_manager.cc:1918] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:11.082865 32305 log.cc:1196] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/73c437b1fa1948d1ae286bf24d11495d
I20260928 03:03:11.083367 32305 ts_tablet_manager.cc:1939] T 73c437b1fa1948d1ae286bf24d11495d P 9911aa7496d04ab382aab49bed9f822d: Deleting consensus metadata
I20260928 03:03:11.084225 31797 catalog_manager.cc:4841] TS 4b5174ef4a7142018cc0a1220254606b (127.12.182.66:45023): tablet 73c437b1fa1948d1ae286bf24d11495d (table test-table [id=6e7fc8469de34537a302fdb50b68ff3f]) successfully deleted
I20260928 03:03:11.085759 31797 catalog_manager.cc:4841] TS 9911aa7496d04ab382aab49bed9f822d (127.12.182.67:44723): tablet 73c437b1fa1948d1ae286bf24d11495d (table test-table [id=6e7fc8469de34537a302fdb50b68ff3f]) successfully deleted
I20260928 03:03:11.067893 32303 raft_consensus.cc:2229] T 73c437b1fa1948d1ae286bf24d11495d P d9fb8fcc98a04c09a43777d35e6486cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:11.173166 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 31872
I20260928 03:03:11.187594 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 32003
W20260928 03:03:11.187677 31799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.182.65:43993 (error 108)
I20260928 03:03:11.190096 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.189550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43993 (local address 127.0.0.1:35376)
0928 03:03:11.189797 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:11.189928 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 03:03:11.190418 31799 catalog_manager.cc:4584] TS d9fb8fcc98a04c09a43777d35e6486cd (127.12.182.65:43993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73c437b1fa1948d1ae286bf24d11495d: Network error: Client connection negotiation failed: client connection to 127.12.182.65:43993: connect: Connection refused (error 111)
I20260928 03:03:11.194567 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.188520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.12.182.66:45023)
0928 03:03:11.188703 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:11.188709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:11.188734 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:03:11.188801 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:11.188806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:11.188905 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:11.189015 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:11.189386 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:11.190367 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:11.191608 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:11.192019 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:11.192133 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:11.193120 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:11.193129 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:11.193142 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:11.193859 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:11.193904 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:11.193924 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:11.194019 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:11.194153 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 03:03:11.194262 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 03:03:11.195211 32125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:11.244117 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.243544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43993 (local address 127.0.0.1:35392)
0928 03:03:11.243820 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:11.243938 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 03:03:11.245170 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.244797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43993 (local address 127.0.0.1:35406)
0928 03:03:11.244946 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:11.245031 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 03:03:11.301280 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 32003
I20260928 03:03:11.305459 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.304960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43993 (local address 127.0.0.1:35410)
0928 03:03:11.305179 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:11.305276 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 03:03:11.306756 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.306355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43993 (local address 127.0.0.1:35418)
0928 03:03:11.306513 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:11.306595 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 03:03:11.317135 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 32133
I20260928 03:03:11.323163 32275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.317858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.12.182.67:44723)
0928 03:03:11.318006 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:11.318011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:11.318029 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:03:11.318197 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:11.318201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:11.318281 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:11.318373 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:11.318722 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:11.319432 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:11.320685 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:11.321059 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:11.321153 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:11.322078 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:11.322086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:11.322100 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:11.322623 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:11.322670 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:11.322689 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:11.322795 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:11.322840 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 03:03:11.322921 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 03:03:11.323771 32255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:11.387311 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.386717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43993 (local address 127.0.0.1:35432)
0928 03:03:11.386979 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:11.387095 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 03:03:11.388478 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:11.388058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:43993 (local address 127.0.0.1:35442)
0928 03:03:11.388268 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:11.388351 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 03:03:11.410104 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 32133
I20260928 03:03:11.423108 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 31782
I20260928 03:03:11.424100 31843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:11.530462 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 31782
2026-09-28T03:03:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2734 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:11 dist-test-slave-k5rg krb5kdc[32314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:11 dist-test-slave-k5rg krb5kdc[32314](info): set up 2 sockets
Sep 28 03:03:11 dist-test-slave-k5rg krb5kdc[32314](info): commencing operation
krb5kdc: starting...
W20260928 03:03:12.537703 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:12 dist-test-slave-k5rg krb5kdc[32314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:03:12.764792 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:43293
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:43293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:13.083916 32327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:13.084187 32327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:13.084319 32327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:13.091488 32327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:13.091600 32327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:13.091684 32327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:13.091761 32327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:13.091825 32327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:13.101711 32327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:43293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:43293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:13.102998 32327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:13.104310 32327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:13.116106 32333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:13.116489 32335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:13.116461 32332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:13.117434 32327 server_base.cc:1034] running on GCE node
I20260928 03:03:13.118232 32327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:13.119601 32327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:13.120800 32327 hybrid_clock.cc:648] HybridClock initialized: now 1790564593120752 us; error 39 us; skew 500 ppm
I20260928 03:03:13.121120 32327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:13 dist-test-slave-k5rg krb5kdc[32314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564593, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:13.124035 32327 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:03:13.125401 32327 webserver.cc:460] Webserver started at http://127.12.182.126:40407/ using document root <none> and password file <none>
I20260928 03:03:13.126050 32327 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:13.126156 32327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:13.126395 32327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:13.129221 32327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "5b6b34d980d448999633430efa44f6a4"
format_stamp: "Formatted at 2026-09-28 03:03:13 on dist-test-slave-k5rg"
server_key: "be7fb5ab4767ede84e8e8602ea323542"
server_key_iv: "1c6d44b13faceb686445f0e1ad1b6854"
server_key_version: "encryptionkey@0"
I20260928 03:03:13.129951 32327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "5b6b34d980d448999633430efa44f6a4"
format_stamp: "Formatted at 2026-09-28 03:03:13 on dist-test-slave-k5rg"
server_key: "be7fb5ab4767ede84e8e8602ea323542"
server_key_iv: "1c6d44b13faceb686445f0e1ad1b6854"
server_key_version: "encryptionkey@0"
I20260928 03:03:13.134795 32327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 03:03:13.138185 32342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:13.139451 32327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:03:13.139603 32327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "5b6b34d980d448999633430efa44f6a4"
format_stamp: "Formatted at 2026-09-28 03:03:13 on dist-test-slave-k5rg"
server_key: "be7fb5ab4767ede84e8e8602ea323542"
server_key_iv: "1c6d44b13faceb686445f0e1ad1b6854"
server_key_version: "encryptionkey@0"
I20260928 03:03:13.139786 32327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:13.159684 32327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:13.163745 32327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:13.164016 32327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:13.181049 32327 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:43293
I20260928 03:03:13.181072 32393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:43293 every 8 connection(s)
I20260928 03:03:13.182406 32327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:13.186134 32394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:13.186486 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 32327
I20260928 03:03:13.186661 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:13.187088 13017 external_mini_cluster.cc:1428] Setting key 94559f816d4dc7c264a4ac28c0181f68
I20260928 03:03:13.199898 32394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4: Bootstrap starting.
Sep 28 03:03:13 dist-test-slave-k5rg krb5kdc[32314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:13.202992 32394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:13.204324 32394 log.cc:824] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:13.207561 32394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4: No bootstrap required, opened a new log
I20260928 03:03:13.211239 32397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:13.188804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.12.182.126:43293)
0928 03:03:13.189646 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:13.189662 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:13.189712 (+ 50us) server_negotiation.cc:407] Connection header received
0928 03:03:13.192354 (+ 2642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:13.192388 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:13.192882 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:13.193320 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:13.194744 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:13.196664 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:13.198044 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:13.199054 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:13.199246 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:13.203029 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:13.203041 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:13.203060 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:13.203105 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:13.206699 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:13.207353 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:13.207362 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:13.207370 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:13.207571 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:13.207994 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:13.207999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:13.208002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:13.208705 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:13.209017 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:13.209382 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 03:03:13.209857 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":331,"threads_started":1}
I20260928 03:03:13.216325 32394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b34d980d448999633430efa44f6a4" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43293 } }
I20260928 03:03:13.216733 32394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:13.216856 32394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6b34d980d448999633430efa44f6a4, State: Initialized, Role: FOLLOWER
I20260928 03:03:13.217401 32394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b34d980d448999633430efa44f6a4" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43293 } }
I20260928 03:03:13.217695 32394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:13.217836 32394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:13.217984 32394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:13.220077 32394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b34d980d448999633430efa44f6a4" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43293 } }
I20260928 03:03:13.220535 32394 leader_election.cc:302] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b6b34d980d448999633430efa44f6a4; no voters:
I20260928 03:03:13.221105 32394 leader_election.cc:288] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:13.221244 32399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:13.221895 32399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [term 1 LEADER]: Becoming Leader. State: Replica: 5b6b34d980d448999633430efa44f6a4, State: Running, Role: LEADER
I20260928 03:03:13.222645 32399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b34d980d448999633430efa44f6a4" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43293 } }
I20260928 03:03:13.223264 32394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:13.225346 32400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b6b34d980d448999633430efa44f6a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b34d980d448999633430efa44f6a4" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43293 } } }
I20260928 03:03:13.226018 32400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:13.225843 32401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b6b34d980d448999633430efa44f6a4. Latest consensus state: current_term: 1 leader_uuid: "5b6b34d980d448999633430efa44f6a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b34d980d448999633430efa44f6a4" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43293 } } }
I20260928 03:03:13.226646 32401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:13.230650 32407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:13.235430 32407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:13.244760 32407 catalog_manager.cc:1347] Generated new cluster ID: 27a5902dbe4c425a8f26b012bc83b2fc
I20260928 03:03:13.244865 32407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:13.257596 32407 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:13.259559 32407 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:13.270869 32407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4: Generated new TSK 0
I20260928 03:03:13.271777 32407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
Entry for principal kudu/127.12.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab.
I20260928 03:03:13.341238 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:43293
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:13.653456 32420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:13.653764 32420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:13.653903 32420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:13.661156 32420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:13.661278 32420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:13.661441 32420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:03:13.671170 32420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.65.keytab
--principal=kudu/127.12.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:43293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:13.672370 32420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:13.673627 32420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:13.685748 32425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:13.685734 32426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:13.685847 32428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:13.686872 32420 server_base.cc:1034] running on GCE node
I20260928 03:03:13.687500 32420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:13.688555 32420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:13.689740 32420 hybrid_clock.cc:648] HybridClock initialized: now 1790564593689671 us; error 65 us; skew 500 ppm
I20260928 03:03:13.690011 32420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:13 dist-test-slave-k5rg krb5kdc[32314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564593, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:13.697562 32420 init.cc:377] Logged in from keytab as kudu/127.12.182.65@KRBTEST.COM (short username kudu)
I20260928 03:03:13.698987 32420 webserver.cc:460] Webserver started at http://127.12.182.65:40439/ using document root <none> and password file <none>
I20260928 03:03:13.699658 32420 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:13.699761 32420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:13.700013 32420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:13.702822 32420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "ab9fd960d75443b4ad7e7d821e8b7418"
format_stamp: "Formatted at 2026-09-28 03:03:13 on dist-test-slave-k5rg"
server_key: "6c3c911f695590abd38e02aebac874c7"
server_key_iv: "8be00bf44e6661b9a21b9fd62cf0a9ce"
server_key_version: "encryptionkey@0"
I20260928 03:03:13.703490 32420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "ab9fd960d75443b4ad7e7d821e8b7418"
format_stamp: "Formatted at 2026-09-28 03:03:13 on dist-test-slave-k5rg"
server_key: "6c3c911f695590abd38e02aebac874c7"
server_key_iv: "8be00bf44e6661b9a21b9fd62cf0a9ce"
server_key_version: "encryptionkey@0"
I20260928 03:03:13.708297 32420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 03:03:13.711642 32435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:13.712864 32420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:03:13.713002 32420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "ab9fd960d75443b4ad7e7d821e8b7418"
format_stamp: "Formatted at 2026-09-28 03:03:13 on dist-test-slave-k5rg"
server_key: "6c3c911f695590abd38e02aebac874c7"
server_key_iv: "8be00bf44e6661b9a21b9fd62cf0a9ce"
server_key_version: "encryptionkey@0"
I20260928 03:03:13.713178 32420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:13.742237 32420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:13.746424 32420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:13.746691 32420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:13.747630 32420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:13.748917 32420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:13.748993 32420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:13.749111 32420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:13.749171 32420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:13.778069 32420 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:33791
I20260928 03:03:13.778116 32547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:33791 every 8 connection(s)
I20260928 03:03:13.779295 32420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:03:13.786428 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 32420
I20260928 03:03:13.786612 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 03:03:13.787149 13017 external_mini_cluster.cc:1428] Setting key 4616bb35437fba81f9a4288490e25eed
Sep 28 03:03:13 dist-test-slave-k5rg krb5kdc[32314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564593, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.65@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:13.798918 32550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:13.781649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:34669 (local address 127.12.182.126:43293)
0928 03:03:13.782082 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:13.782088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:13.783064 (+ 976us) server_negotiation.cc:407] Connection header received
0928 03:03:13.784811 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:13.784815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:13.784879 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:13.784969 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:13.786397 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:13.787452 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:13.789261 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:13.789894 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:13.790015 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:13.793441 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:13.793446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:13.793450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:13.793483 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:13.795916 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:13.796665 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:13.796671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:13.796675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:13.796832 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:13.797213 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:13.797219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:13.797222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:13.797540 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:13.797697 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:13.798353 (+ 656us) server_negotiation.cc:299] Negotiation successful
0928 03:03:13.798577 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":199,"threads_started":1}
I20260928 03:03:13.799769 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:13.782233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:43293 (local address 127.12.182.65:34669)
0928 03:03:13.782913 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:13.782954 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:13.784567 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:13.785150 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:13.785158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:13.785679 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:13.786234 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:13.786251 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:13.787610 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:13.787614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:13.789112 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:13.789125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:13.790084 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:13.790089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:13.790280 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:13.790877 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:13.790899 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:13.793190 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:13.796073 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:13.796082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:13.796102 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:13.796550 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:13.796927 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:13.796930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:13.796932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:13.797115 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:13.797673 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:13.797679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:13.798008 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 03:03:13.798239 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 03:03:13.798635 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":259,"threads_started":1}
I20260928 03:03:13.801282 32548 heartbeater.cc:347] Connected to a master server at 127.12.182.126:43293
I20260928 03:03:13.801625 32548 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:13.802440 32548 heartbeater.cc:511] Master 127.12.182.126:43293 requested a full tablet report, sending...
I20260928 03:03:13.804576 32359 ts_manager.cc:194] Registered new tserver with Master: ab9fd960d75443b4ad7e7d821e8b7418 (127.12.182.65:33791)
I20260928 03:03:13.806056 32359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.65@KRBTEST.COM'} at 127.12.182.65:34669
WARNING: no policy specified for kudu/127.12.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
Entry for principal kudu/127.12.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab.
I20260928 03:03:13.846143 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:43293
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:14.157348 32554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:14.157594 32554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:14.157765 32554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:14.164808 32554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:14.164913 32554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:14.165069 32554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:03:14.174646 32554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.66.keytab
--principal=kudu/127.12.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:43293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:14.175853 32554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:14.177079 32554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:14.190492 32560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:14.190521 32559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:14.190681 32562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:14.191288 32554 server_base.cc:1034] running on GCE node
I20260928 03:03:14.191979 32554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:14.193033 32554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:14.194264 32554 hybrid_clock.cc:648] HybridClock initialized: now 1790564594194210 us; error 46 us; skew 500 ppm
I20260928 03:03:14.194617 32554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:14 dist-test-slave-k5rg krb5kdc[32314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564594, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:14.198165 32554 init.cc:377] Logged in from keytab as kudu/127.12.182.66@KRBTEST.COM (short username kudu)
I20260928 03:03:14.199641 32554 webserver.cc:460] Webserver started at http://127.12.182.66:36583/ using document root <none> and password file <none>
I20260928 03:03:14.200275 32554 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:14.200388 32554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:14.200664 32554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:14.203473 32554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "c0ae99d539c54228870c46744fe97a7c"
format_stamp: "Formatted at 2026-09-28 03:03:14 on dist-test-slave-k5rg"
server_key: "9f2af8da26839751fc4f472714cafee7"
server_key_iv: "b3b64b05fe21599735cc827aa8b7df3d"
server_key_version: "encryptionkey@0"
I20260928 03:03:14.204200 32554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "c0ae99d539c54228870c46744fe97a7c"
format_stamp: "Formatted at 2026-09-28 03:03:14 on dist-test-slave-k5rg"
server_key: "9f2af8da26839751fc4f472714cafee7"
server_key_iv: "b3b64b05fe21599735cc827aa8b7df3d"
server_key_version: "encryptionkey@0"
I20260928 03:03:14.209084 32554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 03:03:14.212391 32569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:14.213693 32554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:03:14.213852 32554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "c0ae99d539c54228870c46744fe97a7c"
format_stamp: "Formatted at 2026-09-28 03:03:14 on dist-test-slave-k5rg"
server_key: "9f2af8da26839751fc4f472714cafee7"
server_key_iv: "b3b64b05fe21599735cc827aa8b7df3d"
server_key_version: "encryptionkey@0"
I20260928 03:03:14.214025 32554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:14.238358 32554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:14.242242 32554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:14.242497 32554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:14.243435 32554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:14.244819 32554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:14.244899 32554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:14.245018 32554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:14.245069 32554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:14.273986 32554 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:43255
I20260928 03:03:14.274039 32681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:43255 every 8 connection(s)
I20260928 03:03:14.275260 32554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:03:14.277842 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 32554
I20260928 03:03:14.278019 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 03:03:14.278559 13017 external_mini_cluster.cc:1428] Setting key b500d2f00ca9bd7bd6656d0d3ee0d4cd
Sep 28 03:03:14 dist-test-slave-k5rg krb5kdc[32314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564594, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.66@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:14.294795 32550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.277671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:37577 (local address 127.12.182.126:43293)
0928 03:03:14.277831 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:14.277836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:14.279250 (+ 1414us) server_negotiation.cc:407] Connection header received
0928 03:03:14.281462 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:14.281468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:14.281548 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:14.281667 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:14.283073 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.283776 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.285258 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.285744 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.285848 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.289094 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:14.289101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:14.289106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:14.289143 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:14.291999 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:14.292658 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:14.292663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:14.292666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:14.292805 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:14.293154 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:14.293158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:14.293161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:14.293464 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:14.293593 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:14.294236 (+ 643us) server_negotiation.cc:299] Negotiation successful
0928 03:03:14.294457 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 03:03:14.295599 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.278322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:43293 (local address 127.12.182.66:37577)
0928 03:03:14.279059 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:14.279101 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:14.281204 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:14.281878 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:14.281888 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:14.282381 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:14.282914 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.282931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.283917 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.283921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.285123 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.285135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.285966 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.285974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.286185 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.286788 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:14.286810 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:14.288871 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:14.292130 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:14.292137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:14.292150 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:14.292554 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:14.292898 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:14.292901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:14.292903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:14.293070 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:14.293572 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:14.293577 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:14.293903 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 03:03:14.294122 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 03:03:14.294498 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":320,"threads_started":1}
I20260928 03:03:14.296962 32682 heartbeater.cc:347] Connected to a master server at 127.12.182.126:43293
I20260928 03:03:14.297293 32682 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:14.298122 32682 heartbeater.cc:511] Master 127.12.182.126:43293 requested a full tablet report, sending...
I20260928 03:03:14.299703 32359 ts_manager.cc:194] Registered new tserver with Master: c0ae99d539c54228870c46744fe97a7c (127.12.182.66:43255)
I20260928 03:03:14.300575 32359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.66@KRBTEST.COM'} at 127.12.182.66:37577
WARNING: no policy specified for kudu/127.12.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
Entry for principal kudu/127.12.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab.
I20260928 03:03:14.332880 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:43293
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:14.643004 32687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:14.643337 32687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:14.643472 32687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:14.650643 32687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:14.650767 32687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:14.650916 32687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:03:14.660568 32687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.67.keytab
--principal=kudu/127.12.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:43293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:14.661969 32687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:14.663287 32687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:14.675927 32692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:14.675976 32693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:14.675926 32695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:14.676852 32687 server_base.cc:1034] running on GCE node
I20260928 03:03:14.677443 32687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:14.678396 32687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:14.679603 32687 hybrid_clock.cc:648] HybridClock initialized: now 1790564594679551 us; error 43 us; skew 500 ppm
I20260928 03:03:14.679956 32687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:14 dist-test-slave-k5rg krb5kdc[32314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564594, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:14.683167 32687 init.cc:377] Logged in from keytab as kudu/127.12.182.67@KRBTEST.COM (short username kudu)
I20260928 03:03:14.684564 32687 webserver.cc:460] Webserver started at http://127.12.182.67:38623/ using document root <none> and password file <none>
I20260928 03:03:14.685227 32687 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:14.685344 32687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:14.685658 32687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:14.688508 32687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "95a48b678684442680546858565d77fb"
format_stamp: "Formatted at 2026-09-28 03:03:14 on dist-test-slave-k5rg"
server_key: "aac529d62acee3b84906f582ab50eb00"
server_key_iv: "a2593486435de7421fff91b7a6f28ef3"
server_key_version: "encryptionkey@0"
I20260928 03:03:14.689231 32687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "95a48b678684442680546858565d77fb"
format_stamp: "Formatted at 2026-09-28 03:03:14 on dist-test-slave-k5rg"
server_key: "aac529d62acee3b84906f582ab50eb00"
server_key_iv: "a2593486435de7421fff91b7a6f28ef3"
server_key_version: "encryptionkey@0"
I20260928 03:03:14.694195 32687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 03:03:14.697414 32702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:14.698776 32687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:03:14.698944 32687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "95a48b678684442680546858565d77fb"
format_stamp: "Formatted at 2026-09-28 03:03:14 on dist-test-slave-k5rg"
server_key: "aac529d62acee3b84906f582ab50eb00"
server_key_iv: "a2593486435de7421fff91b7a6f28ef3"
server_key_version: "encryptionkey@0"
I20260928 03:03:14.699129 32687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:14.711859 32687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:14.715975 32687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:14.716226 32687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:14.717190 32687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:14.718544 32687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:14.718623 32687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:14.718703 32687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:14.718745 32687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:14.748272 32687 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:33025
I20260928 03:03:14.748306 346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:33025 every 8 connection(s)
I20260928 03:03:14.749460 32687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:03:14.754509 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 32687
I20260928 03:03:14.754698 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 03:03:14.755223 13017 external_mini_cluster.cc:1428] Setting key 80ef03fc00e4c992632cdfa8817ac12a
Sep 28 03:03:14 dist-test-slave-k5rg krb5kdc[32314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564594, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.67@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:14.768707 32550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.751928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:49269 (local address 127.12.182.126:43293)
0928 03:03:14.752153 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:14.752159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:14.753159 (+ 1000us) server_negotiation.cc:407] Connection header received
0928 03:03:14.755326 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:14.755334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:14.755429 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:14.755519 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:14.757127 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.758021 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.759540 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.759944 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.760042 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.763470 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:14.763475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:14.763479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:14.763515 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:14.766036 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:14.766714 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:14.766718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:14.766720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:14.766838 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:14.767204 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:14.767208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:14.767209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:14.767466 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:14.767570 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:14.768211 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 03:03:14.768375 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 03:03:14.769924 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.752304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:43293 (local address 127.12.182.67:49269)
0928 03:03:14.752987 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:14.753036 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:14.754981 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:14.755753 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:14.755761 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:14.756344 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:14.756994 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.757012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.758156 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.758160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.759407 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.759421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.760156 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.760161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.760354 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.760985 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:14.761006 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:14.763243 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:14.766150 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:14.766167 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:14.766180 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:14.766596 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:14.766914 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:14.766917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:14.766919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:14.767109 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:14.767575 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:14.767581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:14.767887 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 03:03:14.768133 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 03:03:14.768617 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":267,"threads_started":1}
I20260928 03:03:14.771312 347 heartbeater.cc:347] Connected to a master server at 127.12.182.126:43293
I20260928 03:03:14.771577 347 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:14.772235 347 heartbeater.cc:511] Master 127.12.182.126:43293 requested a full tablet report, sending...
I20260928 03:03:14.773563 32359 ts_manager.cc:194] Registered new tserver with Master: 95a48b678684442680546858565d77fb (127.12.182.67:33025)
I20260928 03:03:14.774346 32359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.182.67@KRBTEST.COM'} at 127.12.182.67:49269
I20260928 03:03:14.782645 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:03:14.799636 32550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.786510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.12.182.126:43293)
0928 03:03:14.786688 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:14.786694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:14.787073 (+ 379us) server_negotiation.cc:407] Connection header received
0928 03:03:14.787240 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:14.787247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:14.787329 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:14.787425 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:14.789120 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.790203 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.791914 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.792557 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.792693 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.795080 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:14.795085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:14.795089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:14.795120 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:14.797349 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:14.798005 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:14.798009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:14.798011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:14.798137 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:14.798415 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:14.798418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:14.798420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:14.798730 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:14.798912 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:14.799226 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 03:03:14.799399 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 03:03:14.803665 32359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 03:03:14.809522 32548 heartbeater.cc:503] Master 127.12.182.126:43293 was elected leader, sending a full tablet report...
W20260928 03:03:14.810366 32359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:03:14.841823 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.822207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.12.182.65:33791)
0928 03:03:14.834345 (+ 12138us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:14.834355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:14.834385 (+ 30us) server_negotiation.cc:407] Connection header received
0928 03:03:14.834483 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:14.834491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:14.834769 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:14.834936 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:14.835515 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.837044 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.839174 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.840588 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.840731 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.840965 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:14.841285 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 03:03:14.841451 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11976,"thread_start_us":11835,"threads_started":1}
I20260928 03:03:14.841912 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.821844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:33791 (local address 127.0.0.1:59934)
0928 03:03:14.832746 (+ 10902us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:14.832799 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:14.832997 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:14.834973 (+ 1976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:14.834981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:14.835019 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:14.835354 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.835365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.837149 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.837153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.839017 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.839030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.840808 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.840815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.841064 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.841118 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 03:03:14.841229 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 03:03:14.841458 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10574,"mutex_wait_us":44,"thread_start_us":13744,"threads_started":1}
I20260928 03:03:14.845496 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.822450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:43255 (local address 127.0.0.1:60364)
0928 03:03:14.835863 (+ 13413us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:14.835883 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:14.836076 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:14.836895 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:14.836900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:14.836922 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:14.839368 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.839380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.840876 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.840881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.842936 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.842951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.844761 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.844766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.844914 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.844948 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 03:03:14.845027 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 03:03:14.845153 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13276,"thread_start_us":10758,"threads_started":1}
I20260928 03:03:14.845579 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.822974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.12.182.66:43255)
0928 03:03:14.834660 (+ 11686us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:14.834668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:14.835974 (+ 1306us) server_negotiation.cc:407] Connection header received
0928 03:03:14.836263 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:14.836270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:14.836481 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:14.836633 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:14.839561 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.840715 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.843147 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.844552 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.844695 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.844871 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:14.845104 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 03:03:14.845262 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11542,"thread_start_us":11417,"threads_started":1}
I20260928 03:03:14.846141 32483 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224 (DEFAULT_TABLE table=test-table [id=770d1b407c62463dbb1ca3d52081b64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:14.847028 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.835960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:33025 (local address 127.0.0.1:53504)
0928 03:03:14.837569 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:14.837585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:14.837716 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:14.840468 (+ 2752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:14.840472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:14.840498 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:14.840884 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.840894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.842819 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.842824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.844639 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.844654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.846324 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.846331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.846490 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.846520 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 03:03:14.846609 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 03:03:14.846727 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1484,"thread_start_us":1336,"threads_started":1}
I20260928 03:03:14.847087 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.836224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.12.182.67:33025)
0928 03:03:14.839928 (+ 3704us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:14.839936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:14.839964 (+ 28us) server_negotiation.cc:407] Connection header received
0928 03:03:14.840051 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:14.840058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:14.840274 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:14.840454 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:14.841091 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.842674 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.844870 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.846145 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.846267 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.846467 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:14.846649 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 03:03:14.846800 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3540,"thread_start_us":3403,"threads_started":1}
I20260928 03:03:14.847445 32483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:14.850638 32750 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224 (DEFAULT_TABLE table=test-table [id=770d1b407c62463dbb1ca3d52081b64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:14.852043 32750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:14.852490 32617 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224 (DEFAULT_TABLE table=test-table [id=770d1b407c62463dbb1ca3d52081b64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:14.853448 32617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:14.861866 365 tablet_bootstrap.cc:492] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: Bootstrap starting.
I20260928 03:03:14.864238 366 tablet_bootstrap.cc:492] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: Bootstrap starting.
I20260928 03:03:14.865515 365 tablet_bootstrap.cc:654] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:14.865662 367 tablet_bootstrap.cc:492] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: Bootstrap starting.
I20260928 03:03:14.866858 365 log.cc:824] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:14.867385 366 tablet_bootstrap.cc:654] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:14.868491 366 log.cc:824] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:14.868633 367 tablet_bootstrap.cc:654] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:14.869808 367 log.cc:824] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:14.871172 366 tablet_bootstrap.cc:492] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: No bootstrap required, opened a new log
I20260928 03:03:14.871176 365 tablet_bootstrap.cc:492] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: No bootstrap required, opened a new log
I20260928 03:03:14.871407 366 ts_tablet_manager.cc:1397] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260928 03:03:14.871445 365 ts_tablet_manager.cc:1397] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 03:03:14.876112 367 tablet_bootstrap.cc:492] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: No bootstrap required, opened a new log
I20260928 03:03:14.876375 367 ts_tablet_manager.cc:1397] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260928 03:03:14.878477 366 raft_consensus.cc:348] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 } } peers { permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 } } peers { permanent_uuid: "c0ae99d539c54228870c46744fe97a7c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43255 } }
I20260928 03:03:14.878818 366 raft_consensus.cc:374] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:14.878913 366 raft_consensus.cc:729] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ae99d539c54228870c46744fe97a7c, State: Initialized, Role: FOLLOWER
I20260928 03:03:14.879472 366 consensus_queue.cc:260] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 } } peers { permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 } } peers { permanent_uuid: "c0ae99d539c54228870c46744fe97a7c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43255 } }
I20260928 03:03:14.880507 32682 heartbeater.cc:503] Master 127.12.182.126:43293 was elected leader, sending a full tablet report...
I20260928 03:03:14.881035 366 ts_tablet_manager.cc:1428] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260928 03:03:14.882673 365 raft_consensus.cc:348] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 } } peers { permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 } } peers { permanent_uuid: "c0ae99d539c54228870c46744fe97a7c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43255 } }
I20260928 03:03:14.883126 365 raft_consensus.cc:374] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:14.883246 365 raft_consensus.cc:729] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9fd960d75443b4ad7e7d821e8b7418, State: Initialized, Role: FOLLOWER
I20260928 03:03:14.884008 365 consensus_queue.cc:260] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 } } peers { permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 } } peers { permanent_uuid: "c0ae99d539c54228870c46744fe97a7c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43255 } }
I20260928 03:03:14.885728 365 ts_tablet_manager.cc:1428] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260928 03:03:14.886616 367 raft_consensus.cc:348] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 } } peers { permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 } } peers { permanent_uuid: "c0ae99d539c54228870c46744fe97a7c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43255 } }
I20260928 03:03:14.887055 367 raft_consensus.cc:374] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:14.887046 371 raft_consensus.cc:482] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:03:14.887212 367 raft_consensus.cc:729] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a48b678684442680546858565d77fb, State: Initialized, Role: FOLLOWER
I20260928 03:03:14.887261 371 raft_consensus.cc:504] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 } } peers { permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 } } peers { permanent_uuid: "c0ae99d539c54228870c46744fe97a7c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43255 } }
I20260928 03:03:14.887938 367 consensus_queue.cc:260] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 } } peers { permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 } } peers { permanent_uuid: "c0ae99d539c54228870c46744fe97a7c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43255 } }
I20260928 03:03:14.889087 371 leader_election.cc:288] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab9fd960d75443b4ad7e7d821e8b7418 (127.12.182.65:33791), 95a48b678684442680546858565d77fb (127.12.182.67:33025)
I20260928 03:03:14.890101 347 heartbeater.cc:503] Master 127.12.182.126:43293 was elected leader, sending a full tablet report...
I20260928 03:03:14.890633 367 ts_tablet_manager.cc:1428] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260928 03:03:14.897768 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.889734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:47909 (local address 127.12.182.65:33791)
0928 03:03:14.889907 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:14.889913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:14.891254 (+ 1341us) server_negotiation.cc:407] Connection header received
0928 03:03:14.891459 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:14.891464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:14.891536 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:14.891617 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:14.892334 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.893316 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.895773 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.896914 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.897060 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.897131 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:14.897421 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 03:03:14.897517 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 03:03:14.897953 374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.889270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:33791 (local address 127.12.182.66:47909)
0928 03:03:14.891128 (+ 1858us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:14.891156 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:14.891294 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:14.891787 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:14.891792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:14.891821 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:14.892169 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.892181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.893438 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.893441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.895612 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.895626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.897105 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.897110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.897266 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.897304 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 03:03:14.897400 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 03:03:14.897527 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":37,"thread_start_us":517,"threads_started":1}
I20260928 03:03:14.897989 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.890948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:37457 (local address 127.12.182.67:33025)
0928 03:03:14.891119 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:14.891126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:14.891916 (+ 790us) server_negotiation.cc:407] Connection header received
0928 03:03:14.892080 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:14.892085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:14.892152 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:14.892244 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:14.892790 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.893765 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.895966 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.897115 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.897249 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.897294 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:14.897651 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 03:03:14.897774 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 03:03:14.898027 375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.890118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:33025 (local address 127.12.182.66:37457)
0928 03:03:14.891816 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:14.891831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:14.891958 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:14.892268 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:14.892272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:14.892296 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:14.892616 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.892629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.893887 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.893892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.895831 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:14.895841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.897310 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.897315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:14.897488 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.897517 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 03:03:14.897592 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 03:03:14.897749 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"thread_start_us":1430,"threads_started":1}
I20260928 03:03:14.899109 32503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcb1b3ae5a44d36b85b1d8cbdcc0224" candidate_uuid: "c0ae99d539c54228870c46744fe97a7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" is_pre_election: true
I20260928 03:03:14.899257 302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcb1b3ae5a44d36b85b1d8cbdcc0224" candidate_uuid: "c0ae99d539c54228870c46744fe97a7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a48b678684442680546858565d77fb" is_pre_election: true
I20260928 03:03:14.904515 32503 raft_consensus.cc:2454] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ae99d539c54228870c46744fe97a7c in term 0.
I20260928 03:03:14.904515 302 raft_consensus.cc:2454] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ae99d539c54228870c46744fe97a7c in term 0.
I20260928 03:03:14.905870 32571 leader_election.cc:302] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab9fd960d75443b4ad7e7d821e8b7418, c0ae99d539c54228870c46744fe97a7c; no voters:
I20260928 03:03:14.906445 371 raft_consensus.cc:2781] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:03:14.906631 371 raft_consensus.cc:482] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:03:14.906730 371 raft_consensus.cc:3037] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:14.908892 371 raft_consensus.cc:504] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 } } peers { permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 } } peers { permanent_uuid: "c0ae99d539c54228870c46744fe97a7c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43255 } }
I20260928 03:03:14.909511 371 leader_election.cc:288] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [CANDIDATE]: Term 1 election: Requested vote from peers ab9fd960d75443b4ad7e7d821e8b7418 (127.12.182.65:33791), 95a48b678684442680546858565d77fb (127.12.182.67:33025)
I20260928 03:03:14.910131 32503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcb1b3ae5a44d36b85b1d8cbdcc0224" candidate_uuid: "c0ae99d539c54228870c46744fe97a7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9fd960d75443b4ad7e7d821e8b7418"
I20260928 03:03:14.910180 302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcb1b3ae5a44d36b85b1d8cbdcc0224" candidate_uuid: "c0ae99d539c54228870c46744fe97a7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a48b678684442680546858565d77fb"
I20260928 03:03:14.910392 302 raft_consensus.cc:3037] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:14.910392 32503 raft_consensus.cc:3037] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:14.913489 32503 raft_consensus.cc:2454] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ae99d539c54228870c46744fe97a7c in term 1.
I20260928 03:03:14.913488 302 raft_consensus.cc:2454] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ae99d539c54228870c46744fe97a7c in term 1.
I20260928 03:03:14.913962 32571 leader_election.cc:302] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a48b678684442680546858565d77fb, c0ae99d539c54228870c46744fe97a7c; no voters:
I20260928 03:03:14.914244 371 raft_consensus.cc:2781] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:14.914655 371 raft_consensus.cc:686] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 1 LEADER]: Becoming Leader. State: Replica: c0ae99d539c54228870c46744fe97a7c, State: Running, Role: LEADER
I20260928 03:03:14.915096 371 consensus_queue.cc:237] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 } } peers { permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 } } peers { permanent_uuid: "c0ae99d539c54228870c46744fe97a7c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43255 } }
I20260928 03:03:14.919903 32359 catalog_manager.cc:5495] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ae99d539c54228870c46744fe97a7c (127.12.182.66). New cstate: current_term: 1 leader_uuid: "c0ae99d539c54228870c46744fe97a7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ae99d539c54228870c46744fe97a7c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 43255 } health_report { overall_health: HEALTHY } } }
I20260928 03:03:14.955488 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.950403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.12.182.66:43255)
0928 03:03:14.950634 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:14.950638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:14.950657 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:03:14.950793 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:14.950797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:14.950883 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:14.951000 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:14.951306 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.952032 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.953291 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:14.953773 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:14.953862 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:14.954364 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:14.954864 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:14.954981 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:14.955135 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 03:03:14.955208 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 03:03:14.963301 302 raft_consensus.cc:1261] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb [term 1 FOLLOWER]: Refusing update from remote peer c0ae99d539c54228870c46744fe97a7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:14.963330 32503 raft_consensus.cc:1261] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418 [term 1 FOLLOWER]: Refusing update from remote peer c0ae99d539c54228870c46744fe97a7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:14.964221 376 consensus_queue.cc:1035] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a48b678684442680546858565d77fb" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:14.964545 371 consensus_queue.cc:1035] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9fd960d75443b4ad7e7d821e8b7418" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 33791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:14.988162 32359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:14.988392 32359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:14.991919 32359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b6b34d980d448999633430efa44f6a4: Sending DeleteTablet for 3 replicas of tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224
I20260928 03:03:14.993055 32617 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:14 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:60364
I20260928 03:03:14.993144 32483 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:14 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:59934
I20260928 03:03:14.993317 32750 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:14 UTC) from {username='kudu', principal='kudu/127.12.182.126@KRBTEST.COM'} at 127.0.0.1:53504
I20260928 03:03:14.994573 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 32420
I20260928 03:03:15.004120 390 tablet_replica.cc:330] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: stopping tablet replica
I20260928 03:03:15.004748 390 raft_consensus.cc:2229] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:15.005400 390 raft_consensus.cc:2258] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:15.006335 389 tablet_replica.cc:330] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: stopping tablet replica
I20260928 03:03:15.008411 390 ts_tablet_manager.cc:1905] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:15.008677 391 tablet_replica.cc:330] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: stopping tablet replica
I20260928 03:03:15.009065 391 raft_consensus.cc:2229] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:15.009557 391 raft_consensus.cc:2258] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:15.010493 389 raft_consensus.cc:2229] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:03:15.011209 389 raft_consensus.cc:2258] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:15.012800 391 ts_tablet_manager.cc:1905] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:15.016335 390 ts_tablet_manager.cc:1918] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:15.016562 390 log.cc:1196] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-0/wal/wals/7fcb1b3ae5a44d36b85b1d8cbdcc0224
I20260928 03:03:15.017585 390 ts_tablet_manager.cc:1939] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P ab9fd960d75443b4ad7e7d821e8b7418: Deleting consensus metadata
I20260928 03:03:15.019505 391 ts_tablet_manager.cc:1918] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:15.019714 391 log.cc:1196] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/7fcb1b3ae5a44d36b85b1d8cbdcc0224
I20260928 03:03:15.020335 391 ts_tablet_manager.cc:1939] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P 95a48b678684442680546858565d77fb: Deleting consensus metadata
I20260928 03:03:15.020283 32344 catalog_manager.cc:4841] TS ab9fd960d75443b4ad7e7d821e8b7418 (127.12.182.65:33791): tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224 (table test-table [id=770d1b407c62463dbb1ca3d52081b64d]) successfully deleted
I20260928 03:03:15.021888 32344 catalog_manager.cc:4841] TS 95a48b678684442680546858565d77fb (127.12.182.67:33025): tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224 (table test-table [id=770d1b407c62463dbb1ca3d52081b64d]) successfully deleted
Sep 28 03:03:15 dist-test-slave-k5rg krb5kdc[32314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.65@KRBTEST.COM
I20260928 03:03:15.023629 389 ts_tablet_manager.cc:1905] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:15.029366 389 ts_tablet_manager.cc:1918] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:15.029547 389 log.cc:1196] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/7fcb1b3ae5a44d36b85b1d8cbdcc0224
I20260928 03:03:15.030105 389 ts_tablet_manager.cc:1939] T 7fcb1b3ae5a44d36b85b1d8cbdcc0224 P c0ae99d539c54228870c46744fe97a7c: Deleting consensus metadata
I20260928 03:03:15.030457 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:14.995534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.12.182.65:33791)
0928 03:03:14.995711 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:14.995717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:15.010748 (+ 15031us) server_negotiation.cc:407] Connection header received
0928 03:03:15.011230 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:15.011236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:15.011332 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:15.011407 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:15.013263 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:15.014519 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:15.016535 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:15.017511 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:15.017777 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:15.023185 (+ 5408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:15.023200 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:15.023218 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:15.023267 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:15.027078 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:15.027897 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:15.027907 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:15.027911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:15.028069 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:15.028521 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:15.028526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:15.028529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:15.029002 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:15.029305 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:15.029669 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 03:03:15.029905 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 03:03:15.031123 32543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:15.031785 32343 catalog_manager.cc:4841] TS c0ae99d539c54228870c46744fe97a7c (127.12.182.66:43255): tablet 7fcb1b3ae5a44d36b85b1d8cbdcc0224 (table test-table [id=770d1b407c62463dbb1ca3d52081b64d]) successfully deleted
I20260928 03:03:15.143492 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 32420
I20260928 03:03:15.164613 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 32554
Sep 28 03:03:15 dist-test-slave-k5rg krb5kdc[32314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.66@KRBTEST.COM
I20260928 03:03:15.180768 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:15.165407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.12.182.66:43255)
0928 03:03:15.165624 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:15.165631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:15.165889 (+ 258us) server_negotiation.cc:407] Connection header received
0928 03:03:15.166162 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:15.166167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:15.166297 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:15.166411 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:15.167686 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:15.168519 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:15.169912 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:15.170359 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:15.170470 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:15.174763 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:15.174778 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:15.174794 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:15.174837 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:15.177830 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:15.178611 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:15.178617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:15.178620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:15.178805 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:15.179190 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:15.179194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:15.179196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:15.179561 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:15.179845 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:15.180210 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 03:03:15.180383 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 03:03:15.181473 32677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:15.302402 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 32554
I20260928 03:03:15.317484 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 32687
Sep 28 03:03:15 dist-test-slave-k5rg krb5kdc[32314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.67@KRBTEST.COM
I20260928 03:03:15.332013 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:15.318227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.12.182.67:33025)
0928 03:03:15.318401 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:15.318405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:15.318424 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:03:15.318574 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:15.318578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:15.318655 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:15.318740 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:15.320029 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:15.320756 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:15.322164 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:15.322533 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:15.322621 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:15.326774 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:15.326787 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:15.326803 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:15.326836 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:15.329433 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:15.330210 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:15.330217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:15.330220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:15.330354 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:15.330714 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:15.330717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:15.330719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:15.331044 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:15.331260 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:15.331541 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 03:03:15.331711 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 03:03:15.332604 342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:15.419884 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 32687
I20260928 03:03:15.433291 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 32327
I20260928 03:03:15.434365 32389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:15.542616 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 32327
2026-09-28T03:03:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4013 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T03:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:15Z Disabled control of system clock
I20260928 03:03:15.611195 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:40527
--webserver_interface=127.12.182.126
--webserver_port=0
--builtin_ntp_servers=127.12.182.84:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:03:15.927120 400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:15.927409 400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:15.927528 400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:15.934819 400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:15.934932 400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:15.934998 400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:15.935089 400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:15.935151 400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:15.944304 400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:40527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:15.945487 400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:15.947183 400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:15.957820 405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:15.957859 408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:15.957892 406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:15.958940 400 server_base.cc:1034] running on GCE node
I20260928 03:03:15.959602 400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:15.960868 400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:15.962057 400 hybrid_clock.cc:648] HybridClock initialized: now 1790564595962001 us; error 48 us; skew 500 ppm
I20260928 03:03:15.962378 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:15.964480 400 webserver.cc:460] Webserver started at http://127.12.182.126:42407/ using document root <none> and password file <none>
I20260928 03:03:15.965099 400 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:15.965185 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:15.965451 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:15.968196 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c"
format_stamp: "Formatted at 2026-09-28 03:03:15 on dist-test-slave-k5rg"
server_key: "c92fc722aa5d7b1881d4a782af3c76fd"
server_key_iv: "de82e98bcd13de6d0609dff9527588ca"
server_key_version: "encryptionkey@0"
I20260928 03:03:15.968930 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c"
format_stamp: "Formatted at 2026-09-28 03:03:15 on dist-test-slave-k5rg"
server_key: "c92fc722aa5d7b1881d4a782af3c76fd"
server_key_iv: "de82e98bcd13de6d0609dff9527588ca"
server_key_version: "encryptionkey@0"
I20260928 03:03:15.973543 400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 03:03:15.976703 414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:15.977934 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 03:03:15.978080 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c"
format_stamp: "Formatted at 2026-09-28 03:03:15 on dist-test-slave-k5rg"
server_key: "c92fc722aa5d7b1881d4a782af3c76fd"
server_key_iv: "de82e98bcd13de6d0609dff9527588ca"
server_key_version: "encryptionkey@0"
I20260928 03:03:15.978262 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:16.003629 400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:16.004443 400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:16.004709 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:16.021801 400 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:40527
I20260928 03:03:16.021865 465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:40527 every 8 connection(s)
I20260928 03:03:16.023039 400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:16.026554 466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:16.032534 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 400
I20260928 03:03:16.032727 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:16.033262 13017 external_mini_cluster.cc:1428] Setting key e305ed0880775132abfe8da885165cd7
I20260928 03:03:16.037395 466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c: Bootstrap starting.
I20260928 03:03:16.040562 466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:16.041637 466 log.cc:824] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:16.044343 466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c: No bootstrap required, opened a new log
I20260928 03:03:16.045954 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:16.035177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.12.182.126:40527)
0928 03:03:16.036064 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:16.036074 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:16.036112 (+ 38us) server_negotiation.cc:407] Connection header received
0928 03:03:16.037783 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:16.037803 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:16.038225 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:16.038546 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:16.039077 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:16.040352 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:16.041868 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:16.042618 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:16.042783 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:16.043823 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:16.043830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:16.043843 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:16.044344 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:16.044419 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:16.044441 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:16.044543 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:16.044760 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 03:03:16.045028 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":303,"threads_started":1}
I20260928 03:03:16.051506 466 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40527 } }
I20260928 03:03:16.051831 466 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:16.051916 466 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcd4c5e51554ad3afc99a02ec66eb4c, State: Initialized, Role: FOLLOWER
I20260928 03:03:16.052434 466 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40527 } }
I20260928 03:03:16.052644 466 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:16.052759 466 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:16.052902 466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:16.055191 466 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40527 } }
I20260928 03:03:16.055629 466 leader_election.cc:302] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebcd4c5e51554ad3afc99a02ec66eb4c; no voters:
I20260928 03:03:16.056145 466 leader_election.cc:288] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:16.056272 471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:16.056905 471 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [term 1 LEADER]: Becoming Leader. State: Replica: ebcd4c5e51554ad3afc99a02ec66eb4c, State: Running, Role: LEADER
I20260928 03:03:16.057433 471 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40527 } }
I20260928 03:03:16.057976 466 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:16.060446 472 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40527 } } }
I20260928 03:03:16.060830 472 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:16.061506 473 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebcd4c5e51554ad3afc99a02ec66eb4c. Latest consensus state: current_term: 1 leader_uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd4c5e51554ad3afc99a02ec66eb4c" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 40527 } } }
I20260928 03:03:16.061931 473 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:16.062237 480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:16.066581 480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:16.074676 480 catalog_manager.cc:1347] Generated new cluster ID: b22abf6feff746c898ce6f0e5e7f67fc
I20260928 03:03:16.074784 480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:16.088331 480 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:16.089537 480 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:16.097563 480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c: Generated new TSK 0
I20260928 03:03:16.098397 480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:16.110242 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.65:0
--local_ip_for_outbound_sockets=127.12.182.65
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40527
--builtin_ntp_servers=127.12.182.84:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:03:16.425980 490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:16.426250 490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:16.426359 490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:16.433241 490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:16.433326 490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:16.433460 490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.65
I20260928 03:03:16.442265 490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.182.65
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.65
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:16.443414 490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:16.445109 490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:16.456873 495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:16.456838 498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:16.456799 496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:16.457567 490 server_base.cc:1034] running on GCE node
I20260928 03:03:16.458243 490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:16.459096 490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:16.460309 490 hybrid_clock.cc:648] HybridClock initialized: now 1790564596460223 us; error 70 us; skew 500 ppm
I20260928 03:03:16.460644 490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:16.462745 490 webserver.cc:460] Webserver started at http://127.12.182.65:42405/ using document root <none> and password file <none>
I20260928 03:03:16.463353 490 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:16.463447 490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:16.463734 490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:16.466373 490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/data/instance:
uuid: "954fe5fa10ef461db5c4480a73d4b985"
format_stamp: "Formatted at 2026-09-28 03:03:16 on dist-test-slave-k5rg"
server_key: "cf8b528d975da5cd17d9ae347f863286"
server_key_iv: "e2b606e81b4104a0001fdb422e4b6001"
server_key_version: "encryptionkey@0"
I20260928 03:03:16.467073 490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance:
uuid: "954fe5fa10ef461db5c4480a73d4b985"
format_stamp: "Formatted at 2026-09-28 03:03:16 on dist-test-slave-k5rg"
server_key: "cf8b528d975da5cd17d9ae347f863286"
server_key_iv: "e2b606e81b4104a0001fdb422e4b6001"
server_key_version: "encryptionkey@0"
I20260928 03:03:16.471606 490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 03:03:16.474726 504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:16.475909 490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 03:03:16.476064 490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/wal
uuid: "954fe5fa10ef461db5c4480a73d4b985"
format_stamp: "Formatted at 2026-09-28 03:03:16 on dist-test-slave-k5rg"
server_key: "cf8b528d975da5cd17d9ae347f863286"
server_key_iv: "e2b606e81b4104a0001fdb422e4b6001"
server_key_version: "encryptionkey@0"
I20260928 03:03:16.476253 490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:16.494174 490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:16.494916 490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:16.495196 490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:16.496078 490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:16.497433 490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:16.497532 490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:16.497665 490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:16.497735 490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:16.525558 490 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.65:40349
I20260928 03:03:16.525626 616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.65:40349 every 8 connection(s)
I20260928 03:03:16.526808 490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/data/info.pb
I20260928 03:03:16.536135 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 490
I20260928 03:03:16.536290 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-0/wal/instance
I20260928 03:03:16.536693 13017 external_mini_cluster.cc:1428] Setting key e5a178a7bd778fe73df3841e55ac18ac
I20260928 03:03:16.538448 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:16.529139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:46127 (local address 127.12.182.126:40527)
0928 03:03:16.529321 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:16.529325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:16.530540 (+ 1215us) server_negotiation.cc:407] Connection header received
0928 03:03:16.532262 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:16.532268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:16.532344 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:16.532431 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:16.533489 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:16.534241 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:16.535721 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:16.536126 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:16.536245 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:16.537631 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:16.537639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:16.537644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:16.537671 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:16.537706 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:16.537721 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:16.537801 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:16.538004 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 03:03:16.538100 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 03:03:16.539318 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.66:0
--local_ip_for_outbound_sockets=127.12.182.66
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40527
--builtin_ntp_servers=127.12.182.84:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:03:16.539362 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:16.529567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40527 (local address 127.12.182.65:46127)
0928 03:03:16.530399 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:16.530431 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:16.531998 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:16.532658 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:16.532666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:16.532794 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:16.533332 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:16.533349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:16.534384 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:16.534388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:16.535584 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:16.535596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:16.536337 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:16.536344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:16.536550 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:16.537219 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:03:16.537222 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:16.537273 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:03:16.537376 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:16.537814 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:16.537826 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:16.537834 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 03:03:16.537982 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 03:03:16.538359 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":245,"threads_started":1}
I20260928 03:03:16.540771 617 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40527
I20260928 03:03:16.541043 617 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:16.541790 617 heartbeater.cc:511] Master 127.12.182.126:40527 requested a full tablet report, sending...
I20260928 03:03:16.543852 431 ts_manager.cc:194] Registered new tserver with Master: 954fe5fa10ef461db5c4480a73d4b985 (127.12.182.65:40349)
I20260928 03:03:16.545737 431 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.65:46127
W20260928 03:03:16.854209 620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:16.854466 620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:16.854588 620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:16.861811 620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:16.861908 620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:16.862056 620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.66
I20260928 03:03:16.870960 620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.182.66
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.66
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:16.872133 620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:16.874020 620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:16.886300 625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:16.886265 628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:16.886327 626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:16.887295 620 server_base.cc:1034] running on GCE node
I20260928 03:03:16.887806 620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:16.888621 620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:16.889809 620 hybrid_clock.cc:648] HybridClock initialized: now 1790564596889760 us; error 42 us; skew 500 ppm
I20260928 03:03:16.890089 620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:16.892148 620 webserver.cc:460] Webserver started at http://127.12.182.66:42055/ using document root <none> and password file <none>
I20260928 03:03:16.892748 620 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:16.892843 620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:16.893118 620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:16.895750 620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/data/instance:
uuid: "43c39dc951e84d44860855cf90d4ae1c"
format_stamp: "Formatted at 2026-09-28 03:03:16 on dist-test-slave-k5rg"
server_key: "5a064bce13539fbd28879b93712b6397"
server_key_iv: "13a4a8ea4dcc1de252f7e892988e50f5"
server_key_version: "encryptionkey@0"
I20260928 03:03:16.896476 620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance:
uuid: "43c39dc951e84d44860855cf90d4ae1c"
format_stamp: "Formatted at 2026-09-28 03:03:16 on dist-test-slave-k5rg"
server_key: "5a064bce13539fbd28879b93712b6397"
server_key_iv: "13a4a8ea4dcc1de252f7e892988e50f5"
server_key_version: "encryptionkey@0"
I20260928 03:03:16.901041 620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 03:03:16.904229 634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:16.905403 620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 03:03:16.905539 620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/wal
uuid: "43c39dc951e84d44860855cf90d4ae1c"
format_stamp: "Formatted at 2026-09-28 03:03:16 on dist-test-slave-k5rg"
server_key: "5a064bce13539fbd28879b93712b6397"
server_key_iv: "13a4a8ea4dcc1de252f7e892988e50f5"
server_key_version: "encryptionkey@0"
I20260928 03:03:16.905755 620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:16.929024 620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:16.929832 620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:16.930109 620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:16.931018 620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:16.932444 620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:16.932538 620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:16.932641 620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:16.932708 620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:16.961728 620 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.66:33483
I20260928 03:03:16.961803 746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.66:33483 every 8 connection(s)
I20260928 03:03:16.962982 620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/data/info.pb
I20260928 03:03:16.972621 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 620
I20260928 03:03:16.972805 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/wal/instance
I20260928 03:03:16.973215 13017 external_mini_cluster.cc:1428] Setting key 702c61e43979b59702adb1b95b0149bd
I20260928 03:03:16.974716 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:16.965399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:34421 (local address 127.12.182.126:40527)
0928 03:03:16.965575 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:16.965579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:16.967107 (+ 1528us) server_negotiation.cc:407] Connection header received
0928 03:03:16.968783 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:16.968787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:16.968853 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:16.968935 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:16.969910 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:16.970612 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:16.972044 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:16.972552 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:16.972675 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:16.973970 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:16.973974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:16.973978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:16.973995 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:16.974033 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:16.974050 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:16.974139 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:16.974387 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 03:03:16.974464 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 03:03:16.975859 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:16.966004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40527 (local address 127.12.182.66:34421)
0928 03:03:16.966978 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:16.967011 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:16.968565 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:16.969099 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:16.969107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:16.969206 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:16.969770 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:16.969787 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:16.970742 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:16.970746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:16.971930 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:16.971942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:16.972757 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:16.972764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:16.973014 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:16.973684 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:03:16.973688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:16.973727 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:03:16.973802 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:16.974155 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:16.974167 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:16.974175 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 03:03:16.974353 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 03:03:16.974699 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":285,"threads_started":1}
I20260928 03:03:16.976188 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.182.67:0
--local_ip_for_outbound_sockets=127.12.182.67
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40527
--builtin_ntp_servers=127.12.182.84:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:03:16.977177 747 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40527
I20260928 03:03:16.977461 747 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:16.978163 747 heartbeater.cc:511] Master 127.12.182.126:40527 requested a full tablet report, sending...
I20260928 03:03:16.979542 431 ts_manager.cc:194] Registered new tserver with Master: 43c39dc951e84d44860855cf90d4ae1c (127.12.182.66:33483)
I20260928 03:03:16.980482 431 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.66:34421
W20260928 03:03:17.287797 750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:17.288051 750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:17.288177 750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:17.295347 750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:17.295442 750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:17.295584 750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.182.67
I20260928 03:03:17.304395 750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.182.67
--webserver_port=0
--tserver_master_addrs=127.12.182.126:40527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.182.67
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:17.305565 750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:17.307211 750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:17.319185 756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:17.319283 755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:17.319290 758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:17.319995 750 server_base.cc:1034] running on GCE node
I20260928 03:03:17.320511 750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:17.321367 750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:17.322592 750 hybrid_clock.cc:648] HybridClock initialized: now 1790564597322531 us; error 43 us; skew 500 ppm
I20260928 03:03:17.322925 750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:03:17.325085 750 webserver.cc:460] Webserver started at http://127.12.182.67:39149/ using document root <none> and password file <none>
I20260928 03:03:17.325758 750 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:17.325860 750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:17.326144 750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:17.328828 750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/data/instance:
uuid: "9d80d4bfa87a4916b6c9612dd042ffea"
format_stamp: "Formatted at 2026-09-28 03:03:17 on dist-test-slave-k5rg"
server_key: "f90ee58f41372c9032dbc0fca3422626"
server_key_iv: "7c6ccacf0ab02ef5fd1a9fd6c4589b8d"
server_key_version: "encryptionkey@0"
I20260928 03:03:17.329579 750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance:
uuid: "9d80d4bfa87a4916b6c9612dd042ffea"
format_stamp: "Formatted at 2026-09-28 03:03:17 on dist-test-slave-k5rg"
server_key: "f90ee58f41372c9032dbc0fca3422626"
server_key_iv: "7c6ccacf0ab02ef5fd1a9fd6c4589b8d"
server_key_version: "encryptionkey@0"
I20260928 03:03:17.334456 750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 03:03:17.337734 764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:17.339015 750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:03:17.339171 750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/wal
uuid: "9d80d4bfa87a4916b6c9612dd042ffea"
format_stamp: "Formatted at 2026-09-28 03:03:17 on dist-test-slave-k5rg"
server_key: "f90ee58f41372c9032dbc0fca3422626"
server_key_iv: "7c6ccacf0ab02ef5fd1a9fd6c4589b8d"
server_key_version: "encryptionkey@0"
I20260928 03:03:17.339375 750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:17.352387 750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:17.353282 750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:17.353580 750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:17.354635 750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:03:17.356338 750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:03:17.356436 750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:17.356547 750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:03:17.356614 750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:17.387051 750 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.67:45141
I20260928 03:03:17.387091 876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.67:45141 every 8 connection(s)
I20260928 03:03:17.388278 750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/data/info.pb
I20260928 03:03:17.388537 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 750
I20260928 03:03:17.388684 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/wal/instance
I20260928 03:03:17.389155 13017 external_mini_cluster.cc:1428] Setting key d324cfa56b1d06ba18f1ead689680c0c
I20260928 03:03:17.399847 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.390834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.67:60411 (local address 127.12.182.126:40527)
0928 03:03:17.390999 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.391003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.392406 (+ 1403us) server_negotiation.cc:407] Connection header received
0928 03:03:17.394378 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.394382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.394450 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.394526 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:17.395510 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.396208 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.397652 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.398040 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.398131 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.399247 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:17.399251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:17.399254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:17.399272 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:17.399303 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:17.399314 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:17.399376 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.399523 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.399603 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 03:03:17.400698 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.391372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:40527 (local address 127.12.182.67:60411)
0928 03:03:17.392247 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.392296 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:17.394133 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:17.394699 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:17.394706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:17.394816 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:17.395348 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.395363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.396332 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.396335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.397517 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.397528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.398204 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.398209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.398397 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.398988 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:03:17.398990 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:17.399025 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:03:17.399095 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:17.399379 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:17.399385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:17.399390 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 03:03:17.399508 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 03:03:17.399791 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":362,"threads_started":1}
I20260928 03:03:17.401875 877 heartbeater.cc:347] Connected to a master server at 127.12.182.126:40527
I20260928 03:03:17.402125 877 heartbeater.cc:464] Registering TS with master...
I20260928 03:03:17.402777 877 heartbeater.cc:511] Master 127.12.182.126:40527 requested a full tablet report, sending...
I20260928 03:03:17.404094 431 ts_manager.cc:194] Registered new tserver with Master: 9d80d4bfa87a4916b6c9612dd042ffea (127.12.182.67:45141)
I20260928 03:03:17.404963 431 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.182.67:60411
I20260928 03:03:17.405519 13017 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:03:17.414795 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.409425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.12.182.126:40527)
0928 03:03:17.409584 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.409590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.410042 (+ 452us) server_negotiation.cc:407] Connection header received
0928 03:03:17.410196 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.410200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.410270 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.410347 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:17.410789 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.411477 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.412842 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.413223 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.413314 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.414218 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:17.414224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:17.414228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:17.414245 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:17.414276 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:17.414287 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:17.414356 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.414443 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.414521 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 03:03:17.418318 431 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:03:17.424964 431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:03:17.446269 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.438056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.12.182.67:45141)
0928 03:03:17.438818 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.438825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.438855 (+ 30us) server_negotiation.cc:407] Connection header received
0928 03:03:17.438946 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.438953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.439155 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.439896 (+ 741us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:17.440136 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.441481 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.443908 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.444976 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.445455 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.445729 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.445794 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.445941 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":458,"threads_started":1}
I20260928 03:03:17.446295 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.436311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:33483 (local address 127.0.0.1:43072)
0928 03:03:17.437218 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.437258 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:17.437425 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:17.438490 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:17.438497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:17.438527 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:17.438942 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.438956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.441342 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.441348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.443433 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.443451 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.445149 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.445156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.445398 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.445495 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 03:03:17.445650 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 03:03:17.445855 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":75,"thread_start_us":491,"threads_started":1}
I20260928 03:03:17.446319 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.437655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:45141 (local address 127.0.0.1:40088)
0928 03:03:17.438176 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.438201 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:17.438309 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:17.439362 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:17.439366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:17.439391 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:17.439708 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.439722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.441621 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.441627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.443692 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.443707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.445156 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.445162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.445397 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.445511 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 03:03:17.445644 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 03:03:17.445863 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":103,"thread_start_us":271,"threads_started":1}
I20260928 03:03:17.446728 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.438211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40349 (local address 127.0.0.1:51250)
0928 03:03:17.439097 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.439116 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:17.439246 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:17.440210 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:17.440226 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:17.440249 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:17.440560 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.440570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.442257 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.442262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.444359 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.444373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.446052 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.446057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.446248 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.446282 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 03:03:17.446350 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 03:03:17.446460 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":656,"threads_started":1}
I20260928 03:03:17.447171 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.438685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.12.182.65:40349)
0928 03:03:17.439682 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.439694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.439716 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:03:17.439803 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.439810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.440014 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.440711 (+ 697us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:17.440861 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.442107 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.444539 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.445872 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.446579 (+ 707us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.446756 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.446815 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.446939 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":749,"threads_started":1}
I20260928 03:03:17.447934 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.436570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.12.182.66:33483)
0928 03:03:17.437859 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.437869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.437899 (+ 30us) server_negotiation.cc:407] Connection header received
0928 03:03:17.437998 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.438005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.438258 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.439158 (+ 900us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:17.439409 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.441207 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.443681 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.444942 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.445073 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.445332 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.447431 (+ 2099us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.447591 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":967,"threads_started":1}
I20260928 03:03:17.450721 682 tablet_service.cc:1467] Processing CreateTablet for tablet f218a7263a134e30b5f484e148a19a04 (DEFAULT_TABLE table=test-table [id=f2b142d5b8d9404b95cbe2dae38d1706]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:17.450795 812 tablet_service.cc:1467] Processing CreateTablet for tablet f218a7263a134e30b5f484e148a19a04 (DEFAULT_TABLE table=test-table [id=f2b142d5b8d9404b95cbe2dae38d1706]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:17.450795 552 tablet_service.cc:1467] Processing CreateTablet for tablet f218a7263a134e30b5f484e148a19a04 (DEFAULT_TABLE table=test-table [id=f2b142d5b8d9404b95cbe2dae38d1706]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:03:17.451900 682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f218a7263a134e30b5f484e148a19a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:17.452054 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f218a7263a134e30b5f484e148a19a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:17.452054 552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f218a7263a134e30b5f484e148a19a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:17.464781 895 tablet_bootstrap.cc:492] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: Bootstrap starting.
I20260928 03:03:17.467530 896 tablet_bootstrap.cc:492] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985: Bootstrap starting.
I20260928 03:03:17.469745 895 tablet_bootstrap.cc:654] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:17.470314 897 tablet_bootstrap.cc:492] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: Bootstrap starting.
I20260928 03:03:17.471020 896 tablet_bootstrap.cc:654] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:17.471186 895 log.cc:824] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:17.472182 896 log.cc:824] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:17.473766 897 tablet_bootstrap.cc:654] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:17.474344 895 tablet_bootstrap.cc:492] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: No bootstrap required, opened a new log
I20260928 03:03:17.474641 895 ts_tablet_manager.cc:1397] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 03:03:17.474809 897 log.cc:824] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:17.475167 896 tablet_bootstrap.cc:492] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985: No bootstrap required, opened a new log
I20260928 03:03:17.475438 896 ts_tablet_manager.cc:1397] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 03:03:17.481454 897 tablet_bootstrap.cc:492] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: No bootstrap required, opened a new log
I20260928 03:03:17.481769 897 ts_tablet_manager.cc:1397] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20260928 03:03:17.485785 896 raft_consensus.cc:348] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } }
I20260928 03:03:17.486239 896 raft_consensus.cc:374] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:17.486340 896 raft_consensus.cc:729] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954fe5fa10ef461db5c4480a73d4b985, State: Initialized, Role: FOLLOWER
I20260928 03:03:17.485785 895 raft_consensus.cc:348] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } }
I20260928 03:03:17.486671 895 raft_consensus.cc:374] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:17.486768 895 raft_consensus.cc:729] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c39dc951e84d44860855cf90d4ae1c, State: Initialized, Role: FOLLOWER
I20260928 03:03:17.487102 896 consensus_queue.cc:260] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } }
I20260928 03:03:17.487313 895 consensus_queue.cc:260] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } }
I20260928 03:03:17.488391 747 heartbeater.cc:503] Master 127.12.182.126:40527 was elected leader, sending a full tablet report...
I20260928 03:03:17.489054 895 ts_tablet_manager.cc:1428] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260928 03:03:17.489579 897 raft_consensus.cc:348] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } }
I20260928 03:03:17.490005 897 raft_consensus.cc:374] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:17.490115 897 raft_consensus.cc:729] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d80d4bfa87a4916b6c9612dd042ffea, State: Initialized, Role: FOLLOWER
I20260928 03:03:17.490671 897 consensus_queue.cc:260] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } }
I20260928 03:03:17.491784 877 heartbeater.cc:503] Master 127.12.182.126:40527 was elected leader, sending a full tablet report...
I20260928 03:03:17.492365 897 ts_tablet_manager.cc:1428] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: Time spent starting tablet: real 0.010s user 0.001s sys 0.008s
I20260928 03:03:17.494349 617 heartbeater.cc:503] Master 127.12.182.126:40527 was elected leader, sending a full tablet report...
I20260928 03:03:17.494848 896 ts_tablet_manager.cc:1428] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985: Time spent starting tablet: real 0.019s user 0.017s sys 0.002s
I20260928 03:03:17.521798 902 raft_consensus.cc:482] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:03:17.522104 902 raft_consensus.cc:504] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } }
I20260928 03:03:17.523207 901 raft_consensus.cc:482] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:03:17.523463 901 raft_consensus.cc:504] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } }
I20260928 03:03:17.523793 902 leader_election.cc:288] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d80d4bfa87a4916b6c9612dd042ffea (127.12.182.67:45141), 954fe5fa10ef461db5c4480a73d4b985 (127.12.182.65:40349)
I20260928 03:03:17.525187 901 leader_election.cc:288] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43c39dc951e84d44860855cf90d4ae1c (127.12.182.66:33483), 9d80d4bfa87a4916b6c9612dd042ffea (127.12.182.67:45141)
W20260928 03:03:17.528867 618 tablet.cc:2369] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:03:17.533670 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.524119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:38537 (local address 127.12.182.67:45141)
0928 03:03:17.524340 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.524345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.524643 (+ 298us) server_negotiation.cc:407] Connection header received
0928 03:03:17.525011 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.525018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.525131 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.525194 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:17.526110 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.527273 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.531178 (+ 3905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.532463 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.532600 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.532688 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.533221 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.533372 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 03:03:17.534402 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.523967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:45141 (local address 127.12.182.66:38537)
0928 03:03:17.524541 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.524558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:17.524818 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:17.525502 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:17.525516 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:17.525545 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:17.525937 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.525949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.527480 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.527486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.530969 (+ 3483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.530988 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.532828 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.532834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.533037 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.533071 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 03:03:17.533925 (+ 854us) client_negotiation.cc:240] Negotiation successful
0928 03:03:17.534084 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":289,"threads_started":1}
I20260928 03:03:17.539446 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f218a7263a134e30b5f484e148a19a04" candidate_uuid: "43c39dc951e84d44860855cf90d4ae1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" is_pre_election: true
I20260928 03:03:17.540009 832 raft_consensus.cc:2454] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43c39dc951e84d44860855cf90d4ae1c in term 0.
I20260928 03:03:17.540715 635 leader_election.cc:302] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c39dc951e84d44860855cf90d4ae1c, 9d80d4bfa87a4916b6c9612dd042ffea; no voters:
I20260928 03:03:17.541069 902 raft_consensus.cc:2781] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:03:17.541188 902 raft_consensus.cc:482] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:03:17.541288 902 raft_consensus.cc:3037] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:17.541319 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.524894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.66:55725 (local address 127.12.182.65:40349)
0928 03:03:17.525515 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.525520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.534741 (+ 9221us) server_negotiation.cc:407] Connection header received
0928 03:03:17.534990 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.534996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.535096 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.535158 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:17.535741 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.536885 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.539176 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.540133 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.540283 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.540358 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.540893 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.541032 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489}
I20260928 03:03:17.541990 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.524612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40349 (local address 127.12.182.66:55725)
0928 03:03:17.534641 (+ 10029us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.534662 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:17.534843 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:17.535364 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:17.535369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:17.535395 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:17.535571 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.535580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.537076 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.537082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.539006 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.539031 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.540462 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.540468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.540675 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.540715 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 03:03:17.541538 (+ 823us) client_negotiation.cc:240] Negotiation successful
0928 03:03:17.541734 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9892,"thread_start_us":13643,"threads_started":1}
I20260928 03:03:17.543769 572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f218a7263a134e30b5f484e148a19a04" candidate_uuid: "43c39dc951e84d44860855cf90d4ae1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954fe5fa10ef461db5c4480a73d4b985" is_pre_election: true
I20260928 03:03:17.544350 572 raft_consensus.cc:2454] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43c39dc951e84d44860855cf90d4ae1c in term 0.
I20260928 03:03:17.544514 902 raft_consensus.cc:504] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } }
I20260928 03:03:17.545207 902 leader_election.cc:288] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [CANDIDATE]: Term 1 election: Requested vote from peers 9d80d4bfa87a4916b6c9612dd042ffea (127.12.182.67:45141), 954fe5fa10ef461db5c4480a73d4b985 (127.12.182.65:40349)
I20260928 03:03:17.545763 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f218a7263a134e30b5f484e148a19a04" candidate_uuid: "43c39dc951e84d44860855cf90d4ae1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d80d4bfa87a4916b6c9612dd042ffea"
I20260928 03:03:17.545958 832 raft_consensus.cc:3037] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:17.546103 572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f218a7263a134e30b5f484e148a19a04" candidate_uuid: "43c39dc951e84d44860855cf90d4ae1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954fe5fa10ef461db5c4480a73d4b985"
I20260928 03:03:17.546299 572 raft_consensus.cc:3037] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:17.549121 832 raft_consensus.cc:2454] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43c39dc951e84d44860855cf90d4ae1c in term 1.
I20260928 03:03:17.549458 572 raft_consensus.cc:2454] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43c39dc951e84d44860855cf90d4ae1c in term 1.
I20260928 03:03:17.549633 635 leader_election.cc:302] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c39dc951e84d44860855cf90d4ae1c, 9d80d4bfa87a4916b6c9612dd042ffea; no voters:
I20260928 03:03:17.549782 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.542240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:58533 (local address 127.12.182.67:45141)
0928 03:03:17.542414 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.542419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.543005 (+ 586us) server_negotiation.cc:407] Connection header received
0928 03:03:17.543296 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.543301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.543367 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.543427 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:17.544345 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.545303 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.547349 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.548601 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.548749 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.548814 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.549368 (+ 554us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.549505 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:03:17.549916 902 raft_consensus.cc:2781] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:17.550347 907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.542304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.67:45141 (local address 127.12.182.65:58533)
0928 03:03:17.542921 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.542937 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:17.543140 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:17.543607 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:17.543612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:17.543634 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:17.544178 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.544192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.545453 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.545458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.547168 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.547181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.548940 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.548947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.549119 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.549152 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 03:03:17.549926 (+ 774us) client_negotiation.cc:240] Negotiation successful
0928 03:03:17.550058 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":325,"threads_started":1}
I20260928 03:03:17.550393 902 raft_consensus.cc:686] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 1 LEADER]: Becoming Leader. State: Replica: 43c39dc951e84d44860855cf90d4ae1c, State: Running, Role: LEADER
I20260928 03:03:17.551211 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f218a7263a134e30b5f484e148a19a04" candidate_uuid: "954fe5fa10ef461db5c4480a73d4b985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" is_pre_election: true
I20260928 03:03:17.551115 902 consensus_queue.cc:237] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } }
I20260928 03:03:17.551503 832 raft_consensus.cc:2379] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 954fe5fa10ef461db5c4480a73d4b985 in current term 1: Already voted for candidate 43c39dc951e84d44860855cf90d4ae1c in this term.
I20260928 03:03:17.552008 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.538522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.182.65:60989 (local address 127.12.182.66:33483)
0928 03:03:17.538692 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.538697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.542382 (+ 3685us) server_negotiation.cc:407] Connection header received
0928 03:03:17.542623 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.542629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.542711 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.542774 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:03:17.544873 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.545829 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.549578 (+ 3749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.551074 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.551219 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.551282 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.551677 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.551798 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 03:03:17.552039 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.525890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.66:33483 (local address 127.12.182.65:60989)
0928 03:03:17.542032 (+ 16142us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.542051 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:17.542480 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:17.542964 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:17.542968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:17.542992 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:03:17.544713 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.544729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.547375 (+ 2646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.547380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.549385 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:17.549397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.551282 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.551289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:17.551456 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.551492 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 03:03:17.551583 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 03:03:17.551717 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16012,"thread_start_us":15865,"threads_started":1}
I20260928 03:03:17.552909 702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f218a7263a134e30b5f484e148a19a04" candidate_uuid: "954fe5fa10ef461db5c4480a73d4b985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c39dc951e84d44860855cf90d4ae1c" is_pre_election: true
I20260928 03:03:17.553855 505 leader_election.cc:302] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954fe5fa10ef461db5c4480a73d4b985; no voters: 43c39dc951e84d44860855cf90d4ae1c, 9d80d4bfa87a4916b6c9612dd042ffea
I20260928 03:03:17.554273 901 raft_consensus.cc:2726] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 03:03:17.556053 430 catalog_manager.cc:5495] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 43c39dc951e84d44860855cf90d4ae1c (127.12.182.66). New cstate: current_term: 1 leader_uuid: "43c39dc951e84d44860855cf90d4ae1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c39dc951e84d44860855cf90d4ae1c" member_type: VOTER last_known_addr { host: "127.12.182.66" port: 33483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 } health_report { overall_health: UNKNOWN } } }
I20260928 03:03:17.599692 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.595058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.12.182.66:33483)
0928 03:03:17.595220 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.595224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.595241 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:03:17.595328 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.595332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.595406 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.595519 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:17.595834 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.596529 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.597838 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.598239 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.598327 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.598713 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:17.599198 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:17.599310 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.599355 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.599414 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:03:17.607533 832 raft_consensus.cc:1261] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [term 1 FOLLOWER]: Refusing update from remote peer 43c39dc951e84d44860855cf90d4ae1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:17.607580 572 raft_consensus.cc:1261] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 1 FOLLOWER]: Refusing update from remote peer 43c39dc951e84d44860855cf90d4ae1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:17.608372 908 consensus_queue.cc:1035] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d80d4bfa87a4916b6c9612dd042ffea" member_type: VOTER last_known_addr { host: "127.12.182.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:17.608744 902 consensus_queue.cc:1035] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954fe5fa10ef461db5c4480a73d4b985" member_type: VOTER last_known_addr { host: "127.12.182.65" port: 40349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:17.619405 911 mvcc.cc:205] Tried to move back new op lower bound from 7334152591790772224 to 7334152591574528000. Current Snapshot: MvccSnapshot[applied={T|T < 7334152591790772224}]
I20260928 03:03:17.631867 430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:17.632105 430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:03:17.635258 430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebcd4c5e51554ad3afc99a02ec66eb4c: Sending DeleteTablet for 3 replicas of tablet f218a7263a134e30b5f484e148a19a04
I20260928 03:03:17.636297 682 tablet_service.cc:1514] Processing DeleteTablet for tablet f218a7263a134e30b5f484e148a19a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:17 UTC) from {username='slave'} at 127.0.0.1:43072
I20260928 03:03:17.636448 812 tablet_service.cc:1514] Processing DeleteTablet for tablet f218a7263a134e30b5f484e148a19a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:17 UTC) from {username='slave'} at 127.0.0.1:40088
I20260928 03:03:17.636858 552 tablet_service.cc:1514] Processing DeleteTablet for tablet f218a7263a134e30b5f484e148a19a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:03:17 UTC) from {username='slave'} at 127.0.0.1:51250
I20260928 03:03:17.638556 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 490
I20260928 03:03:17.670176 922 tablet_replica.cc:330] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: stopping tablet replica
I20260928 03:03:17.670840 922 raft_consensus.cc:2229] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:17.673802 922 raft_consensus.cc:2258] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:17.675165 921 tablet_replica.cc:330] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: stopping tablet replica
I20260928 03:03:17.675609 921 raft_consensus.cc:2229] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:03:17.676267 921 raft_consensus.cc:2258] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:17.676609 923 tablet_replica.cc:330] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985: stopping tablet replica
I20260928 03:03:17.676615 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.639952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.12.182.65:40349)
0928 03:03:17.640116 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.640122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.664807 (+ 24685us) server_negotiation.cc:407] Connection header received
0928 03:03:17.665360 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.665367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.665493 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.665583 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:17.666350 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.667668 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.671298 (+ 3630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.672035 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.672142 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.674839 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:17.674849 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:17.674866 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:17.675631 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:17.675700 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:17.675727 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:17.675835 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.676133 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.676263 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:03:17.677098 922 ts_tablet_manager.cc:1905] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:17.677146 923 raft_consensus.cc:2229] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:03:17.677742 923 raft_consensus.cc:2258] T f218a7263a134e30b5f484e148a19a04 P 954fe5fa10ef461db5c4480a73d4b985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:03:17.678205 612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:17.680117 921 ts_tablet_manager.cc:1905] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:03:17.685046 922 ts_tablet_manager.cc:1918] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:17.685232 922 log.cc:1196] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-2/wal/wals/f218a7263a134e30b5f484e148a19a04
I20260928 03:03:17.685782 922 ts_tablet_manager.cc:1939] T f218a7263a134e30b5f484e148a19a04 P 9d80d4bfa87a4916b6c9612dd042ffea: Deleting consensus metadata
I20260928 03:03:17.687832 415 catalog_manager.cc:4841] TS 9d80d4bfa87a4916b6c9612dd042ffea (127.12.182.67:45141): tablet f218a7263a134e30b5f484e148a19a04 (table test-table [id=f2b142d5b8d9404b95cbe2dae38d1706]) successfully deleted
I20260928 03:03:17.687891 921 ts_tablet_manager.cc:1918] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:03:17.688099 921 log.cc:1196] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: Deleting WAL directory at /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564295078766-13017-0/minicluster-data/ts-1/wal/wals/f218a7263a134e30b5f484e148a19a04
I20260928 03:03:17.688652 921 ts_tablet_manager.cc:1939] T f218a7263a134e30b5f484e148a19a04 P 43c39dc951e84d44860855cf90d4ae1c: Deleting consensus metadata
I20260928 03:03:17.690568 415 catalog_manager.cc:4841] TS 43c39dc951e84d44860855cf90d4ae1c (127.12.182.66:33483): tablet f218a7263a134e30b5f484e148a19a04 (table test-table [id=f2b142d5b8d9404b95cbe2dae38d1706]) successfully deleted
I20260928 03:03:17.777947 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 490
I20260928 03:03:17.799091 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 620
W20260928 03:03:17.799054 415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.182.65:40349 (error 108)
I20260928 03:03:17.800915 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.800345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40349 (local address 127.0.0.1:51270)
0928 03:03:17.800608 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.800748 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260928 03:03:17.801182 415 catalog_manager.cc:4584] TS 954fe5fa10ef461db5c4480a73d4b985 (127.12.182.65:40349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f218a7263a134e30b5f484e148a19a04: Network error: Client connection negotiation failed: client connection to 127.12.182.65:40349: connect: Connection refused (error 111)
I20260928 03:03:17.807689 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.800482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43096 (local address 127.12.182.66:33483)
0928 03:03:17.800662 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.800668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.800688 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:03:17.800905 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.800911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.801038 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.801151 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:17.801804 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.802944 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.804235 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.804947 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.805073 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.806037 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:17.806048 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:17.806071 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:17.806865 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:17.806951 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:17.806976 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:17.807081 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.807236 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.807336 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 03:03:17.808421 742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:17.854776 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.854273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40349 (local address 127.0.0.1:51280)
0928 03:03:17.854485 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.854583 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 03:03:17.855782 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.855439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40349 (local address 127.0.0.1:51296)
0928 03:03:17.855557 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.855629 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 03:03:17.916474 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.915986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40349 (local address 127.0.0.1:51298)
0928 03:03:17.916205 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.916306 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 03:03:17.917500 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.917139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40349 (local address 127.0.0.1:51314)
0928 03:03:17.917303 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.917379 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 03:03:17.920439 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 620
I20260928 03:03:17.935323 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 750
I20260928 03:03:17.941771 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.936029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.12.182.67:45141)
0928 03:03:17.936202 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:17.936209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:17.936265 (+ 56us) server_negotiation.cc:407] Connection header received
0928 03:03:17.936453 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:17.936457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:17.936541 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:17.936634 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:17.937035 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.937764 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.938982 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:17.939388 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:17.939493 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:17.940475 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:17.940483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:17.940498 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:17.941094 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:17.941140 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:17.941159 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:17.941267 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:17.941385 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 03:03:17.941481 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 03:03:17.942373 872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:17.998716 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.998232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40349 (local address 127.0.0.1:51320)
0928 03:03:17.998449 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.998553 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 03:03:17.999701 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:17.999328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.65:40349 (local address 127.0.0.1:51322)
0928 03:03:17.999516 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:17.999583 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.65:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 03:03:18.032348 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 750
I20260928 03:03:18.050047 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 400
I20260928 03:03:18.051155 461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:18.159193 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 400
2026-09-28T03:03:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2615 ms)
[----------] 6 tests from EncryptionPolicyTest (20209 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:18 dist-test-slave-k5rg krb5kdc[932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:18 dist-test-slave-k5rg krb5kdc[932](info): set up 2 sockets
Sep 28 03:03:18 dist-test-slave-k5rg krb5kdc[932](info): commencing operation
krb5kdc: starting...
W20260928 03:03:19.177927 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:19 dist-test-slave-k5rg krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:19.423283 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:19.742271 945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:19.742583 945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:19.742717 945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:19.750000 945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:19.750129 945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:19.750204 945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:19.750281 945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:19.750347 945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:19.761003 945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:19.762351 945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:19.763667 945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:19.775668 950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:19.775615 951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:19.775810 953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:19.776896 945 server_base.cc:1034] running on GCE node
I20260928 03:03:19.777781 945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:19.779213 945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:19.780421 945 hybrid_clock.cc:648] HybridClock initialized: now 1790564599780371 us; error 47 us; skew 500 ppm
I20260928 03:03:19.780747 945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:19 dist-test-slave-k5rg krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:19.787971 945 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:19.789453 945 webserver.cc:460] Webserver started at http://127.0.0.1:39921/ using document root <none> and password file <none>
I20260928 03:03:19.790133 945 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:19.790261 945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:19.790539 945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:19.793429 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "7a016296b9664363af2617a275749a40"
format_stamp: "Formatted at 2026-09-28 03:03:19 on dist-test-slave-k5rg"
server_key: "febd5e9b2869f69cfd292cda1be2a65e"
server_key_iv: "5a5217b86bdf7bf822b5f3a3abd1a833"
server_key_version: "encryptionkey@0"
I20260928 03:03:19.794226 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "7a016296b9664363af2617a275749a40"
format_stamp: "Formatted at 2026-09-28 03:03:19 on dist-test-slave-k5rg"
server_key: "febd5e9b2869f69cfd292cda1be2a65e"
server_key_iv: "5a5217b86bdf7bf822b5f3a3abd1a833"
server_key_version: "encryptionkey@0"
I20260928 03:03:19.799100 945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 03:03:19.802361 960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:19.803606 945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:03:19.803753 945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "7a016296b9664363af2617a275749a40"
format_stamp: "Formatted at 2026-09-28 03:03:19 on dist-test-slave-k5rg"
server_key: "febd5e9b2869f69cfd292cda1be2a65e"
server_key_iv: "5a5217b86bdf7bf822b5f3a3abd1a833"
server_key_version: "encryptionkey@0"
I20260928 03:03:19.803920 945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:19.827183 945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:19.831240 945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:19.831490 945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:19.848503 945 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39515
I20260928 03:03:19.848532 1011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39515 every 8 connection(s)
I20260928 03:03:19.849840 945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:19.853302 1012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:19.855995 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 945
I20260928 03:03:19.856189 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:19.856789 13017 external_mini_cluster.cc:1428] Setting key d49774b10243dcb6d70306f031c88c74
I20260928 03:03:19.865633 1012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40: Bootstrap starting.
I20260928 03:03:19.869076 1012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:19.870078 1012 log.cc:824] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:03:19 dist-test-slave-k5rg krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 03:03:19.872586 1012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40: No bootstrap required, opened a new log
I20260928 03:03:19.879235 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:19.858658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59669 (local address 127.0.0.1:39515)
0928 03:03:19.859380 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:19.859394 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:19.859444 (+ 50us) server_negotiation.cc:407] Connection header received
0928 03:03:19.861518 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:19.861546 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:19.862112 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:19.862528 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:19.863854 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:19.865792 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:19.867138 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:19.868127 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:19.868300 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:19.872068 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:19.872076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:19.872090 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:19.872128 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:19.875131 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:19.875902 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:19.875909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:19.875914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:19.876076 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:19.876519 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:19.876522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:19.876527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:19.877051 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:19.877379 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:19.877893 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 03:03:19.878209 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":276,"threads_started":1}
I20260928 03:03:19.880615 1012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a016296b9664363af2617a275749a40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260928 03:03:19.880975 1012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:19.881067 1012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a016296b9664363af2617a275749a40, State: Initialized, Role: FOLLOWER
I20260928 03:03:19.881690 1012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a016296b9664363af2617a275749a40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260928 03:03:19.882006 1012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:19.882164 1012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:19.882323 1012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:19.883096 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:19.884568 1012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a016296b9664363af2617a275749a40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260928 03:03:19.885016 1012 leader_election.cc:302] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a016296b9664363af2617a275749a40; no voters:
I20260928 03:03:19.885565 1012 leader_election.cc:288] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:19.885744 1017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:19.887431 1012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:19.893918 1017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [term 1 LEADER]: Becoming Leader. State: Replica: 7a016296b9664363af2617a275749a40, State: Running, Role: LEADER
I20260928 03:03:19.894565 1017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a016296b9664363af2617a275749a40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260928 03:03:19.897333 1021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a016296b9664363af2617a275749a40. Latest consensus state: current_term: 1 leader_uuid: "7a016296b9664363af2617a275749a40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a016296b9664363af2617a275749a40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } } }
I20260928 03:03:19.897646 1021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:19.898460 1036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:19.898664 1019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a016296b9664363af2617a275749a40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a016296b9664363af2617a275749a40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } } }
I20260928 03:03:19.898936 1019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:19.898962 1037 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 03:03:19.903733 1036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:19.913121 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:19.895793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42373 (local address 127.0.0.1:39515)
0928 03:03:19.895996 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:19.896007 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:19.896506 (+ 499us) server_negotiation.cc:407] Connection header received
0928 03:03:19.896799 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:19.896818 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:19.896897 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:19.896988 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:19.899377 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:19.900643 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:19.902637 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:19.903442 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:19.903578 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:19.906139 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:19.906145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:19.906149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:19.906184 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:19.909750 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:19.910712 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:19.910719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:19.910722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:19.910866 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:19.911394 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:19.911404 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:19.911408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:19.911900 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:19.912171 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:19.912587 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 03:03:19.912766 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:03:19.913852 1036 catalog_manager.cc:1347] Generated new cluster ID: b1b0085039f74b2a997a376f9a6a9bcc
I20260928 03:03:19.913954 1036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:19.921905 1036 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:19.923089 1036 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:19.928579 1036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a016296b9664363af2617a275749a40: Generated new TSK 0
I20260928 03:03:19.929378 1036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:19.935712 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 945
I20260928 03:03:19.937309 1007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:20.024708 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 945
W20260928 03:03:20.038655 930 connection.cc:535] client connection to 127.0.0.1:39515 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39515): Connection reset by peer (error 104)
2026-09-28T03:03:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1860 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:20 dist-test-slave-k5rg krb5kdc[1044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:20 dist-test-slave-k5rg krb5kdc[1044](info): set up 2 sockets
Sep 28 03:03:20 dist-test-slave-k5rg krb5kdc[1044](info): commencing operation
krb5kdc: starting...
W20260928 03:03:21.049878 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:21 dist-test-slave-k5rg krb5kdc[1044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:21.301249 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:21.613933 1057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:21.614236 1057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:21.614360 1057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:21.621493 1057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:21.621644 1057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:21.621726 1057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:21.621810 1057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:21.621868 1057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:21.631605 1057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:21.632877 1057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:21.634121 1057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:21.645054 1062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:21.645129 1063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:21.645123 1065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:21.646011 1057 server_base.cc:1034] running on GCE node
I20260928 03:03:21.646727 1057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:21.648049 1057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:21.649235 1057 hybrid_clock.cc:648] HybridClock initialized: now 1790564601649188 us; error 37 us; skew 500 ppm
I20260928 03:03:21.649520 1057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:21 dist-test-slave-k5rg krb5kdc[1044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:21.652413 1057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:21.653851 1057 webserver.cc:460] Webserver started at http://127.0.0.1:43217/ using document root <none> and password file <none>
I20260928 03:03:21.654479 1057 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:21.654590 1057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:21.654856 1057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:21.657655 1057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "a65d7b723a4140eaa0328e7272e6d8be"
format_stamp: "Formatted at 2026-09-28 03:03:21 on dist-test-slave-k5rg"
server_key: "32a1f7cab870be834119566944221c3d"
server_key_iv: "c45152dca226ba03dad7ed3c9e0f934c"
server_key_version: "encryptionkey@0"
I20260928 03:03:21.658373 1057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "a65d7b723a4140eaa0328e7272e6d8be"
format_stamp: "Formatted at 2026-09-28 03:03:21 on dist-test-slave-k5rg"
server_key: "32a1f7cab870be834119566944221c3d"
server_key_iv: "c45152dca226ba03dad7ed3c9e0f934c"
server_key_version: "encryptionkey@0"
I20260928 03:03:21.663136 1057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:03:21.666352 1072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:21.667574 1057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:03:21.667726 1057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "a65d7b723a4140eaa0328e7272e6d8be"
format_stamp: "Formatted at 2026-09-28 03:03:21 on dist-test-slave-k5rg"
server_key: "32a1f7cab870be834119566944221c3d"
server_key_iv: "c45152dca226ba03dad7ed3c9e0f934c"
server_key_version: "encryptionkey@0"
I20260928 03:03:21.667896 1057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:21.691311 1057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:21.695375 1057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:21.695636 1057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:21.712826 1057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38151
I20260928 03:03:21.712848 1123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38151 every 8 connection(s)
I20260928 03:03:21.714114 1057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:21.717667 1124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:21.723699 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 1057
I20260928 03:03:21.723902 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:21.724493 13017 external_mini_cluster.cc:1428] Setting key 188bdde0925a94a96b337c436e083617
I20260928 03:03:21.728869 1124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be: Bootstrap starting.
I20260928 03:03:21.731876 1124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:21.732811 1124 log.cc:824] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:21.735849 1124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be: No bootstrap required, opened a new log
I20260928 03:03:21.737829 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:21.726731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59367 (local address 127.0.0.1:38151)
0928 03:03:21.727476 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:21.727487 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:21.727527 (+ 40us) server_negotiation.cc:407] Connection header received
0928 03:03:21.729183 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:21.729204 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:21.729572 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:21.729932 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:21.730441 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:21.731689 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:21.732980 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:21.733920 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:21.734062 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:21.735185 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:21.735196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:21.735215 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:21.735880 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:21.735978 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:21.736008 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:21.736180 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:21.736410 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 03:03:21.736575 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":339,"threads_started":1}
I20260928 03:03:21.741817 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:21.744402 1124 raft_consensus.cc:348] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d7b723a4140eaa0328e7272e6d8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38151 } }
I20260928 03:03:21.744915 1124 raft_consensus.cc:374] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:21.745023 1124 raft_consensus.cc:729] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65d7b723a4140eaa0328e7272e6d8be, State: Initialized, Role: FOLLOWER
I20260928 03:03:21.745570 1124 consensus_queue.cc:260] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d7b723a4140eaa0328e7272e6d8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38151 } }
I20260928 03:03:21.745855 1124 raft_consensus.cc:388] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:21.746025 1124 raft_consensus.cc:482] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:21.746284 1124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:21.749153 1124 raft_consensus.cc:504] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d7b723a4140eaa0328e7272e6d8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38151 } }
I20260928 03:03:21.749826 1124 leader_election.cc:302] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a65d7b723a4140eaa0328e7272e6d8be; no voters:
I20260928 03:03:21.750474 1124 leader_election.cc:288] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:21.750626 1134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:21.752118 1134 raft_consensus.cc:686] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [term 1 LEADER]: Becoming Leader. State: Replica: a65d7b723a4140eaa0328e7272e6d8be, State: Running, Role: LEADER
I20260928 03:03:21.752521 1124 sys_catalog.cc:564] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:21.752749 1134 consensus_queue.cc:237] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d7b723a4140eaa0328e7272e6d8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38151 } }
I20260928 03:03:21.753418 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:21.746064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59089 (local address 127.0.0.1:38151)
0928 03:03:21.746267 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:21.746279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:21.746632 (+ 353us) server_negotiation.cc:407] Connection header received
0928 03:03:21.746817 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:21.746821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:21.746884 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:21.746964 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:21.747575 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:21.748372 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:21.750148 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:21.750861 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:21.751000 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:21.752577 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:21.752583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:21.752588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:21.752611 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:21.752680 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:21.752704 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:21.752827 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:21.753042 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 03:03:21.753096 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 03:03:21.755837 1136 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [sys.catalog]: SysCatalogTable state changed. Reason: New leader a65d7b723a4140eaa0328e7272e6d8be. Latest consensus state: current_term: 1 leader_uuid: "a65d7b723a4140eaa0328e7272e6d8be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d7b723a4140eaa0328e7272e6d8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38151 } } }
I20260928 03:03:21.756232 1136 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:21.757827 1135 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a65d7b723a4140eaa0328e7272e6d8be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65d7b723a4140eaa0328e7272e6d8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38151 } } }
I20260928 03:03:21.758119 1135 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:21.758677 1141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:21.763442 1141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:21.774067 1141 catalog_manager.cc:1347] Generated new cluster ID: cddb5268deb24f049c964377b24212be
I20260928 03:03:21.774197 1141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:21.801340 1141 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:21.803155 1141 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:21.812880 1141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a65d7b723a4140eaa0328e7272e6d8be: Generated new TSK 0
I20260928 03:03:21.813941 1141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:21.821740 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1057
W20260928 03:03:21.822904 1119 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59367
W20260928 03:03:21.823673 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1057: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:21.823863 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 1057
W20260928 03:03:21.837463 1042 connection.cc:535] client connection to 127.0.0.1:38151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38151): Connection reset by peer (error 104)
2026-09-28T03:03:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1798 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:21 dist-test-slave-k5rg krb5kdc[1155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:21 dist-test-slave-k5rg krb5kdc[1155](info): set up 2 sockets
Sep 28 03:03:21 dist-test-slave-k5rg krb5kdc[1155](info): commencing operation
krb5kdc: starting...
W20260928 03:03:22.849773 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:22 dist-test-slave-k5rg krb5kdc[1155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:23.077950 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:23.397250 1168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:23.397518 1168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:23.397667 1168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:23.404721 1168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:23.404824 1168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:23.404891 1168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:23.404969 1168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:23.405027 1168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:23.414618 1168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:23.415844 1168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:23.417035 1168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:23.427748 1173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:23.427838 1176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:23.427780 1174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:23.428947 1168 server_base.cc:1034] running on GCE node
I20260928 03:03:23.429674 1168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:23.430989 1168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:23.432196 1168 hybrid_clock.cc:648] HybridClock initialized: now 1790564603432147 us; error 39 us; skew 500 ppm
I20260928 03:03:23.432520 1168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:23 dist-test-slave-k5rg krb5kdc[1155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:23.435453 1168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:23.436870 1168 webserver.cc:460] Webserver started at http://127.0.0.1:38819/ using document root <none> and password file <none>
I20260928 03:03:23.437505 1168 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:23.437626 1168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:23.437909 1168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:23.440706 1168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "8aa712af3b684d75859ed329d9bd6c0f"
format_stamp: "Formatted at 2026-09-28 03:03:23 on dist-test-slave-k5rg"
server_key: "cd145ebc3bda8d0cc0583a0a7430768e"
server_key_iv: "ffd11a4cb74e8ae2004407ef6ab9e0ff"
server_key_version: "encryptionkey@0"
I20260928 03:03:23.441411 1168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "8aa712af3b684d75859ed329d9bd6c0f"
format_stamp: "Formatted at 2026-09-28 03:03:23 on dist-test-slave-k5rg"
server_key: "cd145ebc3bda8d0cc0583a0a7430768e"
server_key_iv: "ffd11a4cb74e8ae2004407ef6ab9e0ff"
server_key_version: "encryptionkey@0"
I20260928 03:03:23.446214 1168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:03:23.449489 1183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:23.450793 1168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:03:23.450941 1168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "8aa712af3b684d75859ed329d9bd6c0f"
format_stamp: "Formatted at 2026-09-28 03:03:23 on dist-test-slave-k5rg"
server_key: "cd145ebc3bda8d0cc0583a0a7430768e"
server_key_iv: "ffd11a4cb74e8ae2004407ef6ab9e0ff"
server_key_version: "encryptionkey@0"
I20260928 03:03:23.451109 1168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:23.456498 1168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:23.460330 1168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:23.460579 1168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:23.477543 1168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44013
I20260928 03:03:23.477581 1234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44013 every 8 connection(s)
I20260928 03:03:23.478941 1168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:23.480490 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 1168
I20260928 03:03:23.480674 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:23.481197 13017 external_mini_cluster.cc:1428] Setting key e73e749611f0a726ea7210205e1a5ca4
I20260928 03:03:23.483290 1235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:23.488353 1238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:23.482869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42671 (local address 127.0.0.1:44013)
0928 03:03:23.483611 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:23.483627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:23.483668 (+ 41us) server_negotiation.cc:407] Connection header received
0928 03:03:23.485181 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:23.485201 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:23.485515 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:23.485870 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:23.486885 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:23.486890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:23.486902 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:23.486919 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:23.486997 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:23.487016 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:23.487179 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:23.487277 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 03:03:23.487439 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":310,"threads_started":1}
I20260928 03:03:23.491847 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:23.494735 1235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f: Bootstrap starting.
I20260928 03:03:23.498245 1238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:23.496080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40787 (local address 127.0.0.1:44013)
0928 03:03:23.496242 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:23.496248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:23.496524 (+ 276us) server_negotiation.cc:407] Connection header received
0928 03:03:23.496683 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:23.496688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:23.496733 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:23.496807 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:23.497672 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:23.497677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:23.497682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:23.497697 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:23.497733 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:23.497747 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:23.497836 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:23.497939 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 03:03:23.497993 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 03:03:23.498281 1235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:23.499413 1235 log.cc:824] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:23.501953 1235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f: No bootstrap required, opened a new log
I20260928 03:03:23.509045 1235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa712af3b684d75859ed329d9bd6c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44013 } }
I20260928 03:03:23.509397 1235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:23.509486 1235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aa712af3b684d75859ed329d9bd6c0f, State: Initialized, Role: FOLLOWER
I20260928 03:03:23.510103 1235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa712af3b684d75859ed329d9bd6c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44013 } }
I20260928 03:03:23.510310 1235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:23.510439 1235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:23.510587 1235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:23.512625 1235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa712af3b684d75859ed329d9bd6c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44013 } }
I20260928 03:03:23.513123 1235 leader_election.cc:302] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aa712af3b684d75859ed329d9bd6c0f; no voters:
I20260928 03:03:23.513769 1235 leader_election.cc:288] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:23.513878 1245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:23.514477 1245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [term 1 LEADER]: Becoming Leader. State: Replica: 8aa712af3b684d75859ed329d9bd6c0f, State: Running, Role: LEADER
I20260928 03:03:23.515018 1245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa712af3b684d75859ed329d9bd6c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44013 } }
I20260928 03:03:23.515736 1235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:23.517645 1246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aa712af3b684d75859ed329d9bd6c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa712af3b684d75859ed329d9bd6c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44013 } } }
I20260928 03:03:23.518060 1246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:23.518712 1247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aa712af3b684d75859ed329d9bd6c0f. Latest consensus state: current_term: 1 leader_uuid: "8aa712af3b684d75859ed329d9bd6c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa712af3b684d75859ed329d9bd6c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44013 } } }
I20260928 03:03:23.519073 1247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:23.519594 1252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:23.524711 1252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:23.533185 1252 catalog_manager.cc:1347] Generated new cluster ID: 286a7f43ec84491fbe5a789480b90f32
I20260928 03:03:23.533305 1252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:23.551209 1252 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:23.551676 1252 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:23.564383 1252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aa712af3b684d75859ed329d9bd6c0f: Generated new TSK 0
I20260928 03:03:23.565398 1252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:23.579751 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1168
W20260928 03:03:23.581203 1230 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42671
W20260928 03:03:23.581847 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1168: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:23.582000 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 1168
2026-09-28T03:03:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1758 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:23 dist-test-slave-k5rg krb5kdc[1266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:23 dist-test-slave-k5rg krb5kdc[1266](info): set up 2 sockets
Sep 28 03:03:23 dist-test-slave-k5rg krb5kdc[1266](info): commencing operation
krb5kdc: starting...
W20260928 03:03:24.605767 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:24 dist-test-slave-k5rg krb5kdc[1266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:24.832716 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:25.145530 1279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:25.145867 1279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:25.145995 1279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:25.153110 1279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:25.153220 1279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:25.153293 1279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:25.153378 1279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:25.153436 1279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:25.164178 1279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:25.165426 1279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:25.166778 1279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:25.177891 1287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:25.177878 1284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:25.178381 1285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:25.178983 1279 server_base.cc:1034] running on GCE node
I20260928 03:03:25.179694 1279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:25.181087 1279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:25.182281 1279 hybrid_clock.cc:648] HybridClock initialized: now 1790564605182228 us; error 42 us; skew 500 ppm
I20260928 03:03:25.182588 1279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:25 dist-test-slave-k5rg krb5kdc[1266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:25.194015 1279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:25.195508 1279 webserver.cc:460] Webserver started at http://127.0.0.1:38593/ using document root <none> and password file <none>
I20260928 03:03:25.196189 1279 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:25.196302 1279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:25.196586 1279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:25.199582 1279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "672cac7a3a4d428ba5dc88c134eb755d"
format_stamp: "Formatted at 2026-09-28 03:03:25 on dist-test-slave-k5rg"
server_key: "f76f29360f9745addd324007bdbe0e8d"
server_key_iv: "bd4869fb4a2fe8a0dd48d6922001b847"
server_key_version: "encryptionkey@0"
I20260928 03:03:25.200340 1279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "672cac7a3a4d428ba5dc88c134eb755d"
format_stamp: "Formatted at 2026-09-28 03:03:25 on dist-test-slave-k5rg"
server_key: "f76f29360f9745addd324007bdbe0e8d"
server_key_iv: "bd4869fb4a2fe8a0dd48d6922001b847"
server_key_version: "encryptionkey@0"
I20260928 03:03:25.205567 1279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 03:03:25.209527 1294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:25.210894 1279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:03:25.211043 1279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "672cac7a3a4d428ba5dc88c134eb755d"
format_stamp: "Formatted at 2026-09-28 03:03:25 on dist-test-slave-k5rg"
server_key: "f76f29360f9745addd324007bdbe0e8d"
server_key_iv: "bd4869fb4a2fe8a0dd48d6922001b847"
server_key_version: "encryptionkey@0"
I20260928 03:03:25.211216 1279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:25.245173 1279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:25.249082 1279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:25.249348 1279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:25.266187 1279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46149
I20260928 03:03:25.266225 1345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46149 every 8 connection(s)
I20260928 03:03:25.267513 1279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:25.271106 1346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:25.274768 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 1279
I20260928 03:03:25.274951 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:25.275491 13017 external_mini_cluster.cc:1428] Setting key dd45031c25bd6f87f7186a2d979424a7
I20260928 03:03:25.283634 1346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d: Bootstrap starting.
I20260928 03:03:25.287313 1346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:25.288585 1346 log.cc:824] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:03:25 dist-test-slave-k5rg krb5kdc[1266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 03:03:25.291457 1346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d: No bootstrap required, opened a new log
I20260928 03:03:25.299093 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:25.277286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45435 (local address 127.0.0.1:46149)
0928 03:03:25.278041 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:25.278053 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:25.278093 (+ 40us) server_negotiation.cc:407] Connection header received
0928 03:03:25.279704 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:25.279725 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:25.280118 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:25.280441 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:25.282177 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:25.283532 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:25.285233 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:25.285973 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:25.286099 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:25.290884 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:25.290893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:25.290907 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:25.290941 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:25.294727 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:25.295434 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:25.295442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:25.295450 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:25.295664 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:25.295984 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:25.295989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:25.295992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:25.296671 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:25.296966 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:25.297309 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 03:03:25.297776 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":365,"threads_started":1}
I20260928 03:03:25.301182 1346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672cac7a3a4d428ba5dc88c134eb755d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260928 03:03:25.301549 1346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:25.301681 1346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672cac7a3a4d428ba5dc88c134eb755d, State: Initialized, Role: FOLLOWER
I20260928 03:03:25.302253 1346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672cac7a3a4d428ba5dc88c134eb755d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260928 03:03:25.302492 1346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:25.302631 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:25.302626 1346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:25.302788 1346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:25.305790 1346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672cac7a3a4d428ba5dc88c134eb755d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260928 03:03:25.306388 1346 leader_election.cc:302] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 672cac7a3a4d428ba5dc88c134eb755d; no voters:
I20260928 03:03:25.306983 1346 leader_election.cc:288] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:25.307072 1355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:25.307646 1355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [term 1 LEADER]: Becoming Leader. State: Replica: 672cac7a3a4d428ba5dc88c134eb755d, State: Running, Role: LEADER
I20260928 03:03:25.308307 1355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672cac7a3a4d428ba5dc88c134eb755d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260928 03:03:25.310293 1355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 672cac7a3a4d428ba5dc88c134eb755d. Latest consensus state: current_term: 1 leader_uuid: "672cac7a3a4d428ba5dc88c134eb755d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672cac7a3a4d428ba5dc88c134eb755d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } } }
I20260928 03:03:25.311843 1355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:25.311489 1357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "672cac7a3a4d428ba5dc88c134eb755d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672cac7a3a4d428ba5dc88c134eb755d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } } }
I20260928 03:03:25.312193 1357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:25.315289 1346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:25.316661 1360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:25.322732 1360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:25.336028 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:25.308284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35779 (local address 127.0.0.1:46149)
0928 03:03:25.308475 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:25.308481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:25.319073 (+ 10592us) server_negotiation.cc:407] Connection header received
0928 03:03:25.319289 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:25.319295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:25.319376 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:25.319495 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:25.321623 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:25.322810 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:25.324698 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:25.325443 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:25.325557 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:25.328808 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:25.328816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:25.328821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:25.328864 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:25.332769 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:25.333575 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:25.333581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:25.333584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:25.333764 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:25.334235 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:25.334243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:25.334247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:25.334623 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:25.334852 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:25.335430 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 03:03:25.335650 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 03:03:25.343228 1360 catalog_manager.cc:1347] Generated new cluster ID: 75d3ce5870334bf4a87e8fde59b4eff0
I20260928 03:03:25.343353 1360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:25.362989 1360 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:25.364846 1360 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:25.377092 1360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 672cac7a3a4d428ba5dc88c134eb755d: Generated new TSK 0
I20260928 03:03:25.377882 1360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:25.384523 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1279
I20260928 03:03:25.385737 1341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:25.475361 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 1279
2026-09-28T03:03:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:25 dist-test-slave-k5rg krb5kdc[1378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:25 dist-test-slave-k5rg krb5kdc[1378](info): set up 2 sockets
Sep 28 03:03:25 dist-test-slave-k5rg krb5kdc[1378](info): commencing operation
krb5kdc: starting...
W20260928 03:03:26.485828 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:26 dist-test-slave-k5rg krb5kdc[1378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:26.709389 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:27.024538 1391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:27.024838 1391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:27.024956 1391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:27.032223 1391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:27.032344 1391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:27.032399 1391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:27.032483 1391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:27.032536 1391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:27.042553 1391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:27.043926 1391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:27.045192 1391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:27.056759 1396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:27.056732 1397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:27.056936 1399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:27.057752 1391 server_base.cc:1034] running on GCE node
I20260928 03:03:27.058506 1391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:27.059850 1391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:27.061053 1391 hybrid_clock.cc:648] HybridClock initialized: now 1790564607061011 us; error 27 us; skew 500 ppm
I20260928 03:03:27.061388 1391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:27 dist-test-slave-k5rg krb5kdc[1378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:27.064443 1391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:27.065974 1391 webserver.cc:460] Webserver started at http://127.0.0.1:34341/ using document root <none> and password file <none>
I20260928 03:03:27.066632 1391 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:27.066749 1391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:27.067028 1391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:27.070056 1391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "0208afb0691248e29efc203afb141208"
format_stamp: "Formatted at 2026-09-28 03:03:27 on dist-test-slave-k5rg"
server_key: "16532e37268efb3f172b9607112c09ce"
server_key_iv: "3f2fbe6ea6444c6e20e1af7503c75643"
server_key_version: "encryptionkey@0"
I20260928 03:03:27.070803 1391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "0208afb0691248e29efc203afb141208"
format_stamp: "Formatted at 2026-09-28 03:03:27 on dist-test-slave-k5rg"
server_key: "16532e37268efb3f172b9607112c09ce"
server_key_iv: "3f2fbe6ea6444c6e20e1af7503c75643"
server_key_version: "encryptionkey@0"
I20260928 03:03:27.075986 1391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:03:27.079599 1406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:27.080996 1391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:03:27.081158 1391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "0208afb0691248e29efc203afb141208"
format_stamp: "Formatted at 2026-09-28 03:03:27 on dist-test-slave-k5rg"
server_key: "16532e37268efb3f172b9607112c09ce"
server_key_iv: "3f2fbe6ea6444c6e20e1af7503c75643"
server_key_version: "encryptionkey@0"
I20260928 03:03:27.081382 1391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:27.110301 1391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:27.114315 1391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:27.114578 1391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:27.131709 1391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38043
I20260928 03:03:27.131736 1457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38043 every 8 connection(s)
I20260928 03:03:27.133065 1391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:27.136667 1458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:27.139989 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 1391
I20260928 03:03:27.140168 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:27.140723 13017 external_mini_cluster.cc:1428] Setting key 3c79041d0ca4d1153d01bc2d3b0623e4
I20260928 03:03:27.148767 1458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208: Bootstrap starting.
I20260928 03:03:27.152189 1458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:27.153314 1458 log.cc:824] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:27.153744 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:27.142545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57357 (local address 127.0.0.1:38043)
0928 03:03:27.143271 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:27.143282 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:27.143322 (+ 40us) server_negotiation.cc:407] Connection header received
0928 03:03:27.144948 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:27.144969 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:27.145370 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:27.145724 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:27.146262 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:27.147675 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:27.149443 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:27.150137 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:27.150275 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:27.151440 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:27.151448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:27.151460 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:27.152007 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:27.152085 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:27.152113 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:27.152204 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:27.152444 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 03:03:27.152753 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":316,"threads_started":1}
I20260928 03:03:27.155962 1458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208: No bootstrap required, opened a new log
I20260928 03:03:27.157169 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:27.166340 1458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0208afb0691248e29efc203afb141208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38043 } }
I20260928 03:03:27.166721 1458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:27.166822 1458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0208afb0691248e29efc203afb141208, State: Initialized, Role: FOLLOWER
I20260928 03:03:27.167412 1458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0208afb0691248e29efc203afb141208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38043 } }
I20260928 03:03:27.167657 1458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:27.167821 1458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:27.168012 1458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:27.169433 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:27.162863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40599 (local address 127.0.0.1:38043)
0928 03:03:27.163069 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:27.163076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:27.163946 (+ 870us) server_negotiation.cc:407] Connection header received
0928 03:03:27.164080 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:27.164085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:27.164168 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:27.164287 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:27.164735 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:27.165822 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:27.167167 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:27.167593 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:27.167757 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:27.168850 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:27.168854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:27.168858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:27.168875 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:27.168915 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:27.168933 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:27.169003 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:27.169119 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 03:03:27.169195 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 03:03:27.170933 1458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0208afb0691248e29efc203afb141208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38043 } }
I20260928 03:03:27.171407 1458 leader_election.cc:302] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0208afb0691248e29efc203afb141208; no voters:
I20260928 03:03:27.171928 1458 leader_election.cc:288] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:27.172043 1468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:27.172657 1468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [term 1 LEADER]: Becoming Leader. State: Replica: 0208afb0691248e29efc203afb141208, State: Running, Role: LEADER
I20260928 03:03:27.173255 1468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0208afb0691248e29efc203afb141208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38043 } }
I20260928 03:03:27.174006 1458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:27.175652 1470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0208afb0691248e29efc203afb141208. Latest consensus state: current_term: 1 leader_uuid: "0208afb0691248e29efc203afb141208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0208afb0691248e29efc203afb141208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38043 } } }
I20260928 03:03:27.176105 1470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:27.176043 1469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0208afb0691248e29efc203afb141208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0208afb0691248e29efc203afb141208" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38043 } } }
I20260928 03:03:27.176321 1469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:27.178187 1473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:27.183382 1473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:27.192477 1473 catalog_manager.cc:1347] Generated new cluster ID: bf746844c170495fbb45a3f51d6b698b
I20260928 03:03:27.192585 1473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:27.222308 1473 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:27.223928 1473 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:27.235456 1473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0208afb0691248e29efc203afb141208: Generated new TSK 0
I20260928 03:03:27.236217 1473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:27.244405 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1391
W20260928 03:03:27.245549 1453 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57357
W20260928 03:03:27.246192 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1391: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:27.246313 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 1391
2026-09-28T03:03:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1770 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:27 dist-test-slave-k5rg krb5kdc[1489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:27 dist-test-slave-k5rg krb5kdc[1489](info): set up 2 sockets
Sep 28 03:03:27 dist-test-slave-k5rg krb5kdc[1489](info): commencing operation
krb5kdc: starting...
W20260928 03:03:28.254119 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:28 dist-test-slave-k5rg krb5kdc[1489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:28.482637 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:28.800863 1502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:28.801146 1502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:28.801307 1502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:28.808555 1502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:28.808674 1502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:28.808753 1502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:28.808831 1502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:28.808887 1502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:28.818744 1502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:28.819986 1502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:28.821220 1502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:28.832219 1508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:28.832293 1510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:28.832247 1507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:28.832944 1502 server_base.cc:1034] running on GCE node
I20260928 03:03:28.833712 1502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:28.835155 1502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:28.836364 1502 hybrid_clock.cc:648] HybridClock initialized: now 1790564608836311 us; error 49 us; skew 500 ppm
I20260928 03:03:28.836689 1502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:28 dist-test-slave-k5rg krb5kdc[1489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:28.839694 1502 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:28.841053 1502 webserver.cc:460] Webserver started at http://127.0.0.1:36405/ using document root <none> and password file <none>
I20260928 03:03:28.841831 1502 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:28.841953 1502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:28.842232 1502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:28.845206 1502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "4480f3946ae04b529b543030df0516da"
format_stamp: "Formatted at 2026-09-28 03:03:28 on dist-test-slave-k5rg"
server_key: "c2834f779741848edec9dc534171c880"
server_key_iv: "dc8d31453837393aaaf6f9d7a0b2899e"
server_key_version: "encryptionkey@0"
I20260928 03:03:28.845959 1502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "4480f3946ae04b529b543030df0516da"
format_stamp: "Formatted at 2026-09-28 03:03:28 on dist-test-slave-k5rg"
server_key: "c2834f779741848edec9dc534171c880"
server_key_iv: "dc8d31453837393aaaf6f9d7a0b2899e"
server_key_version: "encryptionkey@0"
I20260928 03:03:28.850771 1502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:03:28.853909 1517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:28.855125 1502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 03:03:28.855274 1502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "4480f3946ae04b529b543030df0516da"
format_stamp: "Formatted at 2026-09-28 03:03:28 on dist-test-slave-k5rg"
server_key: "c2834f779741848edec9dc534171c880"
server_key_iv: "dc8d31453837393aaaf6f9d7a0b2899e"
server_key_version: "encryptionkey@0"
I20260928 03:03:28.855455 1502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:28.860790 1502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:28.864707 1502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:28.864960 1502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:28.881420 1502 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46465
I20260928 03:03:28.881461 1568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46465 every 8 connection(s)
I20260928 03:03:28.882814 1502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:28.883884 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 1502
I20260928 03:03:28.884068 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:28.884609 13017 external_mini_cluster.cc:1428] Setting key e8a9655dbd6baea4f4e3f6796b5be2aa
I20260928 03:03:28.887210 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:28.891924 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:28.886397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58345 (local address 127.0.0.1:46465)
0928 03:03:28.887169 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:28.887180 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:28.887224 (+ 44us) server_negotiation.cc:407] Connection header received
0928 03:03:28.888823 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:28.888844 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:28.889161 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:28.889488 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:28.890473 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:28.890479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:28.890494 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:28.890513 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:28.890584 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:28.890618 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:28.890782 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:28.890878 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 03:03:28.891040 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":362,"threads_started":1}
I20260928 03:03:28.895352 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:28.898983 1569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da: Bootstrap starting.
I20260928 03:03:28.902454 1569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:28.902541 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:28.900352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56133 (local address 127.0.0.1:46465)
0928 03:03:28.900533 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:28.900539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:28.900920 (+ 381us) server_negotiation.cc:407] Connection header received
0928 03:03:28.901040 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:28.901044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:28.901099 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:28.901195 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:28.901978 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:28.901983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:28.901987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:28.902003 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:28.902046 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:28.902066 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:28.902175 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:28.902217 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 03:03:28.902269 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 03:03:28.903483 1569 log.cc:824] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:28.905907 1569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da: No bootstrap required, opened a new log
I20260928 03:03:28.912771 1569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480f3946ae04b529b543030df0516da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46465 } }
I20260928 03:03:28.913120 1569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:28.913205 1569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4480f3946ae04b529b543030df0516da, State: Initialized, Role: FOLLOWER
I20260928 03:03:28.913754 1569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480f3946ae04b529b543030df0516da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46465 } }
I20260928 03:03:28.913966 1569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:28.914094 1569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:28.914242 1569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:28.916299 1569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480f3946ae04b529b543030df0516da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46465 } }
I20260928 03:03:28.916760 1569 leader_election.cc:302] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4480f3946ae04b529b543030df0516da; no voters:
I20260928 03:03:28.917305 1569 leader_election.cc:288] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:28.917428 1579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:28.918495 1579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [term 1 LEADER]: Becoming Leader. State: Replica: 4480f3946ae04b529b543030df0516da, State: Running, Role: LEADER
I20260928 03:03:28.918999 1579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480f3946ae04b529b543030df0516da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46465 } }
I20260928 03:03:28.919376 1569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:28.921492 1580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4480f3946ae04b529b543030df0516da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480f3946ae04b529b543030df0516da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46465 } } }
I20260928 03:03:28.921469 1581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4480f3946ae04b529b543030df0516da. Latest consensus state: current_term: 1 leader_uuid: "4480f3946ae04b529b543030df0516da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480f3946ae04b529b543030df0516da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46465 } } }
I20260928 03:03:28.921988 1580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:28.922006 1581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:28.924189 1585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:28.928973 1585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:28.938120 1585 catalog_manager.cc:1347] Generated new cluster ID: 3d0e6b7625e848008b3c133470ab31c3
I20260928 03:03:28.938220 1585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:28.960392 1585 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:28.960825 1585 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:28.966732 1585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4480f3946ae04b529b543030df0516da: Generated new TSK 0
I20260928 03:03:28.967645 1585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:28.983500 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1502
W20260928 03:03:28.984583 1564 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58345
W20260928 03:03:28.985278 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1502: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:28.985427 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 1502
2026-09-28T03:03:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1738 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:29 dist-test-slave-k5rg krb5kdc[1600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:29 dist-test-slave-k5rg krb5kdc[1600](info): set up 2 sockets
Sep 28 03:03:29 dist-test-slave-k5rg krb5kdc[1600](info): commencing operation
krb5kdc: starting...
W20260928 03:03:29.989876 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:30 dist-test-slave-k5rg krb5kdc[1600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:30.217868 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:30.524760 1613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:30.525019 1613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:30.525149 1613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:30.532155 1613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:30.532258 1613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:30.532332 1613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:30.532404 1613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:30.532462 1613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:30.542053 1613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:30.543278 1613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:30.544559 1613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:30.555397 1619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:30.555507 1618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:30.555490 1621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:30.556334 1613 server_base.cc:1034] running on GCE node
I20260928 03:03:30.557053 1613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:30.558388 1613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:30.559576 1613 hybrid_clock.cc:648] HybridClock initialized: now 1790564610559539 us; error 30 us; skew 500 ppm
I20260928 03:03:30.559878 1613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:30 dist-test-slave-k5rg krb5kdc[1600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:30.566633 1613 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:30.567955 1613 webserver.cc:460] Webserver started at http://127.0.0.1:33229/ using document root <none> and password file <none>
I20260928 03:03:30.568569 1613 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:30.568662 1613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:30.568907 1613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:30.571771 1613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "dfb4c58087934b05ab5e40f2d3519642"
format_stamp: "Formatted at 2026-09-28 03:03:30 on dist-test-slave-k5rg"
server_key: "6caed8d163c25440bd9930289ab193a9"
server_key_iv: "752a1482683c96ea2d5659213647013b"
server_key_version: "encryptionkey@0"
I20260928 03:03:30.572474 1613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "dfb4c58087934b05ab5e40f2d3519642"
format_stamp: "Formatted at 2026-09-28 03:03:30 on dist-test-slave-k5rg"
server_key: "6caed8d163c25440bd9930289ab193a9"
server_key_iv: "752a1482683c96ea2d5659213647013b"
server_key_version: "encryptionkey@0"
I20260928 03:03:30.577328 1613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 03:03:30.580554 1628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:30.581810 1613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 03:03:30.581940 1613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "dfb4c58087934b05ab5e40f2d3519642"
format_stamp: "Formatted at 2026-09-28 03:03:30 on dist-test-slave-k5rg"
server_key: "6caed8d163c25440bd9930289ab193a9"
server_key_iv: "752a1482683c96ea2d5659213647013b"
server_key_version: "encryptionkey@0"
I20260928 03:03:30.582091 1613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:30.604189 1613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:30.607805 1613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:30.608022 1613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:30.624842 1613 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43939
I20260928 03:03:30.624876 1679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43939 every 8 connection(s)
I20260928 03:03:30.626185 1613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:30.627703 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 1613
I20260928 03:03:30.627853 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:30.628283 13017 external_mini_cluster.cc:1428] Setting key 4684f2fb49e87e6a97b31a02b09bb983
I20260928 03:03:30.630060 1680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 03:03:30 dist-test-slave-k5rg krb5kdc[1600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 03:03:30.642391 1680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642: Bootstrap starting.
I20260928 03:03:30.646095 1680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:30.647246 1680 log.cc:824] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:30.650570 1680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642: No bootstrap required, opened a new log
I20260928 03:03:30.650800 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:30.629937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:46913 (local address 127.0.0.1:43939)
0928 03:03:30.630710 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:30.630724 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:30.630772 (+ 48us) server_negotiation.cc:407] Connection header received
0928 03:03:30.632826 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:30.632852 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:30.633356 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:30.633774 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:30.634992 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:30.636940 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:30.638123 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:30.639102 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:30.639288 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:30.642991 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:30.643001 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:30.643018 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:30.643063 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:30.646692 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:30.647405 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:30.647413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:30.647419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:30.647617 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:30.648008 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:30.648013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:30.648017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:30.648571 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:30.648860 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:30.649259 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 03:03:30.649563 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":318,"threads_started":1}
I20260928 03:03:30.654410 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:30.659488 1680 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb4c58087934b05ab5e40f2d3519642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260928 03:03:30.659904 1680 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:30.660010 1680 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb4c58087934b05ab5e40f2d3519642, State: Initialized, Role: FOLLOWER
I20260928 03:03:30.660598 1680 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb4c58087934b05ab5e40f2d3519642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260928 03:03:30.660871 1680 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:30.661005 1680 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:30.661168 1680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:30.663985 1680 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb4c58087934b05ab5e40f2d3519642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260928 03:03:30.664463 1680 leader_election.cc:302] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfb4c58087934b05ab5e40f2d3519642; no voters:
I20260928 03:03:30.665079 1680 leader_election.cc:288] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:30.665189 1690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:30.665911 1690 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [term 1 LEADER]: Becoming Leader. State: Replica: dfb4c58087934b05ab5e40f2d3519642, State: Running, Role: LEADER
I20260928 03:03:30.666466 1690 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb4c58087934b05ab5e40f2d3519642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260928 03:03:30.667645 1680 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:30.669375 1691 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfb4c58087934b05ab5e40f2d3519642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb4c58087934b05ab5e40f2d3519642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } } }
I20260928 03:03:30.669871 1691 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:30.670244 1692 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfb4c58087934b05ab5e40f2d3519642. Latest consensus state: current_term: 1 leader_uuid: "dfb4c58087934b05ab5e40f2d3519642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb4c58087934b05ab5e40f2d3519642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } } }
I20260928 03:03:30.670529 1692 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:30.676406 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:30.659372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:47701 (local address 127.0.0.1:43939)
0928 03:03:30.659549 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:30.659555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:30.660043 (+ 488us) server_negotiation.cc:407] Connection header received
0928 03:03:30.660205 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:30.660208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:30.660271 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:30.660381 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:30.662049 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:30.663179 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:30.664412 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:30.665050 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:30.665199 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:30.668253 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:30.668258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:30.668263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:30.668307 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:30.672277 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:30.673298 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:30.673304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:30.673308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:30.673474 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:30.674024 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:30.674029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:30.674033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:30.674420 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:30.675613 (+ 1193us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:30.675783 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 03:03:30.675969 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 03:03:30.678165 1695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:30.683039 1695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:30.691524 1695 catalog_manager.cc:1347] Generated new cluster ID: 3fdb5bc96d4c4ce787c775dabfbbf737
I20260928 03:03:30.691622 1695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:30.708266 1695 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:30.709445 1695 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:30.718104 1695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfb4c58087934b05ab5e40f2d3519642: Generated new TSK 0
I20260928 03:03:30.718863 1695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:30.724126 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1613
I20260928 03:03:30.725919 1675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:30.815910 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 1613
W20260928 03:03:30.829944 1598 connection.cc:535] client connection to 127.0.0.1:43939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43939): Connection reset by peer (error 104)
2026-09-28T03:03:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1831 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:30 dist-test-slave-k5rg krb5kdc[1712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:30 dist-test-slave-k5rg krb5kdc[1712](info): set up 2 sockets
Sep 28 03:03:30 dist-test-slave-k5rg krb5kdc[1712](info): commencing operation
krb5kdc: starting...
W20260928 03:03:31.837816 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:31 dist-test-slave-k5rg krb5kdc[1712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:32.064338 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:32.373939 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:32.374228 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:32.374366 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:32.381418 1725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:32.381541 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:32.381645 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:32.381727 1725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:32.381788 1725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:32.391340 1725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:32.392665 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:32.393908 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:32.404785 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:32.404810 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:32.404785 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:32.405577 1725 server_base.cc:1034] running on GCE node
I20260928 03:03:32.406322 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:32.407665 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:32.408880 1725 hybrid_clock.cc:648] HybridClock initialized: now 1790564612408826 us; error 43 us; skew 500 ppm
I20260928 03:03:32.409202 1725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:32 dist-test-slave-k5rg krb5kdc[1712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:32.412374 1725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:32.413779 1725 webserver.cc:460] Webserver started at http://127.0.0.1:39911/ using document root <none> and password file <none>
I20260928 03:03:32.414387 1725 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:32.414499 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:32.414772 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:32.417539 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "e692295942aa4cc7ae160f9b70c98220"
format_stamp: "Formatted at 2026-09-28 03:03:32 on dist-test-slave-k5rg"
server_key: "17d4b522cdafbd167726c8131dc9892a"
server_key_iv: "f37bbb6e1ea944352cb9cf92e67cbd6f"
server_key_version: "encryptionkey@0"
I20260928 03:03:32.418305 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "e692295942aa4cc7ae160f9b70c98220"
format_stamp: "Formatted at 2026-09-28 03:03:32 on dist-test-slave-k5rg"
server_key: "17d4b522cdafbd167726c8131dc9892a"
server_key_iv: "f37bbb6e1ea944352cb9cf92e67cbd6f"
server_key_version: "encryptionkey@0"
I20260928 03:03:32.423075 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:03:32.426390 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:32.427605 1725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:03:32.427772 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "e692295942aa4cc7ae160f9b70c98220"
format_stamp: "Formatted at 2026-09-28 03:03:32 on dist-test-slave-k5rg"
server_key: "17d4b522cdafbd167726c8131dc9892a"
server_key_iv: "f37bbb6e1ea944352cb9cf92e67cbd6f"
server_key_version: "encryptionkey@0"
I20260928 03:03:32.427950 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:32.441628 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:32.445518 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:32.445806 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:32.462947 1725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42069
I20260928 03:03:32.462970 1791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42069 every 8 connection(s)
I20260928 03:03:32.464272 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:32.465186 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 1725
I20260928 03:03:32.465368 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:32.465943 13017 external_mini_cluster.cc:1428] Setting key 3dfe9f08e785973c5d0ce23937e3a300
I20260928 03:03:32.468631 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:32.478600 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:32.468066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:52529 (local address 127.0.0.1:42069)
0928 03:03:32.468949 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:32.468960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:32.469000 (+ 40us) server_negotiation.cc:407] Connection header received
0928 03:03:32.470608 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:32.470629 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:32.471022 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:32.471378 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:32.471896 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:32.473174 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:32.474462 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:32.475083 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:32.475237 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:32.476436 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:32.476444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:32.476456 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:32.477003 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:32.477091 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:32.477116 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:32.477240 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:32.477509 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 03:03:32.477695 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":483,"threads_started":1}
I20260928 03:03:32.479058 1792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220: Bootstrap starting.
I20260928 03:03:32.482064 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:32.482177 1792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:32.483080 1792 log.cc:824] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:32.485774 1792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220: No bootstrap required, opened a new log
I20260928 03:03:32.492861 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:32.486253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:33107 (local address 127.0.0.1:42069)
0928 03:03:32.486438 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:32.486444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:32.486834 (+ 390us) server_negotiation.cc:407] Connection header received
0928 03:03:32.487022 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:32.487026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:32.487081 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:32.487148 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:32.487754 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:32.488523 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:32.490434 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:32.490904 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:32.490998 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:32.492139 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:32.492144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:32.492147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:32.492162 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:32.492224 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:32.492241 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:32.492336 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:32.492549 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 03:03:32.492599 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 03:03:32.495314 1792 raft_consensus.cc:348] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e692295942aa4cc7ae160f9b70c98220" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } }
I20260928 03:03:32.495667 1792 raft_consensus.cc:374] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:32.495764 1792 raft_consensus.cc:729] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e692295942aa4cc7ae160f9b70c98220, State: Initialized, Role: FOLLOWER
I20260928 03:03:32.496282 1792 consensus_queue.cc:260] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e692295942aa4cc7ae160f9b70c98220" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } }
I20260928 03:03:32.496487 1792 raft_consensus.cc:388] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:32.496620 1792 raft_consensus.cc:482] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:32.496766 1792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:32.499253 1792 raft_consensus.cc:504] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e692295942aa4cc7ae160f9b70c98220" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } }
I20260928 03:03:32.499708 1792 leader_election.cc:302] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e692295942aa4cc7ae160f9b70c98220; no voters:
I20260928 03:03:32.500247 1792 leader_election.cc:288] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:32.500322 1802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:32.500939 1802 raft_consensus.cc:686] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [term 1 LEADER]: Becoming Leader. State: Replica: e692295942aa4cc7ae160f9b70c98220, State: Running, Role: LEADER
I20260928 03:03:32.501458 1802 consensus_queue.cc:237] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e692295942aa4cc7ae160f9b70c98220" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } }
I20260928 03:03:32.502328 1792 sys_catalog.cc:564] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:32.504124 1803 sys_catalog.cc:455] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e692295942aa4cc7ae160f9b70c98220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e692295942aa4cc7ae160f9b70c98220" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } } }
I20260928 03:03:32.504180 1804 sys_catalog.cc:455] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e692295942aa4cc7ae160f9b70c98220. Latest consensus state: current_term: 1 leader_uuid: "e692295942aa4cc7ae160f9b70c98220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e692295942aa4cc7ae160f9b70c98220" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } } }
I20260928 03:03:32.504470 1803 sys_catalog.cc:458] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:32.504504 1804 sys_catalog.cc:458] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:32.505926 1807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:32.510026 1807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:32.517745 1807 catalog_manager.cc:1347] Generated new cluster ID: 6b1e530384e548e98817da3ace96cad5
I20260928 03:03:32.517843 1807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:32.538178 1807 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:32.539494 1807 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:32.546778 1807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e692295942aa4cc7ae160f9b70c98220: Generated new TSK 0
I20260928 03:03:32.547501 1807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:32.554716 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1725
W20260928 03:03:32.555795 1787 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:52529
W20260928 03:03:32.556507 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:32.556620 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 1725
W20260928 03:03:32.570605 1710 connection.cc:535] client connection to 127.0.0.1:42069 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42069): Connection reset by peer (error 104)
2026-09-28T03:03:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1740 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:32 dist-test-slave-k5rg krb5kdc[1823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:32 dist-test-slave-k5rg krb5kdc[1823](info): set up 2 sockets
Sep 28 03:03:32 dist-test-slave-k5rg krb5kdc[1823](info): commencing operation
krb5kdc: starting...
W20260928 03:03:33.557907 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:33 dist-test-slave-k5rg krb5kdc[1823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:33.788343 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:34.104528 1836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:34.104807 1836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:34.104940 1836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:34.112000 1836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:34.112102 1836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:34.112177 1836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:34.112259 1836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:34.112316 1836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:34.121912 1836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:34.123087 1836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:34.124360 1836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:34.135337 1841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:34.135353 1842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:34.135408 1844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:34.136029 1836 server_base.cc:1034] running on GCE node
I20260928 03:03:34.136766 1836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:34.138146 1836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:34.139351 1836 hybrid_clock.cc:648] HybridClock initialized: now 1790564614139306 us; error 37 us; skew 500 ppm
I20260928 03:03:34.139706 1836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:34 dist-test-slave-k5rg krb5kdc[1823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:34.142593 1836 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:34.143886 1836 webserver.cc:460] Webserver started at http://127.0.0.1:34311/ using document root <none> and password file <none>
I20260928 03:03:34.144486 1836 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:34.144583 1836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:34.144819 1836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:34.147650 1836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "6da31b4e8adc4d7eb4d837b018442ee2"
format_stamp: "Formatted at 2026-09-28 03:03:34 on dist-test-slave-k5rg"
server_key: "3313f0c22db28a4aa8c9c8560d578eb2"
server_key_iv: "fc79963237ba10bd96d0f77981b3b3ea"
server_key_version: "encryptionkey@0"
I20260928 03:03:34.148331 1836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "6da31b4e8adc4d7eb4d837b018442ee2"
format_stamp: "Formatted at 2026-09-28 03:03:34 on dist-test-slave-k5rg"
server_key: "3313f0c22db28a4aa8c9c8560d578eb2"
server_key_iv: "fc79963237ba10bd96d0f77981b3b3ea"
server_key_version: "encryptionkey@0"
I20260928 03:03:34.153231 1836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 03:03:34.156649 1851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:34.157912 1836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:03:34.158058 1836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "6da31b4e8adc4d7eb4d837b018442ee2"
format_stamp: "Formatted at 2026-09-28 03:03:34 on dist-test-slave-k5rg"
server_key: "3313f0c22db28a4aa8c9c8560d578eb2"
server_key_iv: "fc79963237ba10bd96d0f77981b3b3ea"
server_key_version: "encryptionkey@0"
I20260928 03:03:34.158237 1836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:34.163446 1836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:34.167383 1836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:34.167629 1836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:34.184006 1836 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41891
I20260928 03:03:34.184018 1902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41891 every 8 connection(s)
I20260928 03:03:34.185551 1836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:34.189356 1903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:34.190554 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 1836
I20260928 03:03:34.190728 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:34.191126 13017 external_mini_cluster.cc:1428] Setting key 1939dae80798a06082e3e27c277da498
I20260928 03:03:34.198333 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:34.192696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:38947 (local address 127.0.0.1:41891)
0928 03:03:34.193418 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:34.193429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:34.193467 (+ 38us) server_negotiation.cc:407] Connection header received
0928 03:03:34.195077 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:34.195124 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:34.195506 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:34.195819 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:34.196833 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:34.196839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:34.196851 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:34.196865 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:34.196923 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:34.196945 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:34.197095 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:34.197247 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 03:03:34.197428 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":294,"threads_started":1}
I20260928 03:03:34.202045 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:34.202379 1903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2: Bootstrap starting.
I20260928 03:03:34.206174 1903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:34.207422 1903 log.cc:824] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:34.209482 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:34.207210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:39903 (local address 127.0.0.1:41891)
0928 03:03:34.207383 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:34.207389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:34.207684 (+ 295us) server_negotiation.cc:407] Connection header received
0928 03:03:34.207850 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:34.207853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:34.207905 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:34.208004 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:34.208896 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:34.208901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:34.208904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:34.208918 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:34.208953 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:34.208968 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:34.209048 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:34.209194 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 03:03:34.209237 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 03:03:34.210569 1903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2: No bootstrap required, opened a new log
I20260928 03:03:34.217813 1903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da31b4e8adc4d7eb4d837b018442ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41891 } }
I20260928 03:03:34.218142 1903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:34.218215 1903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da31b4e8adc4d7eb4d837b018442ee2, State: Initialized, Role: FOLLOWER
I20260928 03:03:34.218699 1903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da31b4e8adc4d7eb4d837b018442ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41891 } }
I20260928 03:03:34.218895 1903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:34.218988 1903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:34.219120 1903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:34.221388 1903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da31b4e8adc4d7eb4d837b018442ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41891 } }
I20260928 03:03:34.221902 1903 leader_election.cc:302] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6da31b4e8adc4d7eb4d837b018442ee2; no voters:
I20260928 03:03:34.222417 1903 leader_election.cc:288] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:34.222536 1913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:34.223120 1913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [term 1 LEADER]: Becoming Leader. State: Replica: 6da31b4e8adc4d7eb4d837b018442ee2, State: Running, Role: LEADER
I20260928 03:03:34.223655 1913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da31b4e8adc4d7eb4d837b018442ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41891 } }
I20260928 03:03:34.224403 1903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:34.226145 1915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6da31b4e8adc4d7eb4d837b018442ee2. Latest consensus state: current_term: 1 leader_uuid: "6da31b4e8adc4d7eb4d837b018442ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da31b4e8adc4d7eb4d837b018442ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41891 } } }
I20260928 03:03:34.226290 1914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6da31b4e8adc4d7eb4d837b018442ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6da31b4e8adc4d7eb4d837b018442ee2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41891 } } }
I20260928 03:03:34.226583 1914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:34.226583 1915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:34.227526 1920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:34.233120 1920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:34.242455 1920 catalog_manager.cc:1347] Generated new cluster ID: 8e47641c03d246098e9bd4a691a0bf6f
I20260928 03:03:34.242556 1920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:34.256392 1920 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:34.256812 1920 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:34.268044 1920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6da31b4e8adc4d7eb4d837b018442ee2: Generated new TSK 0
I20260928 03:03:34.268836 1920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:34.275909 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1836
W20260928 03:03:34.276916 1898 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:38947
W20260928 03:03:34.277525 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1836: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:34.277673 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 1836
2026-09-28T03:03:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1720 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:34 dist-test-slave-k5rg krb5kdc[1934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:34 dist-test-slave-k5rg krb5kdc[1934](info): set up 2 sockets
Sep 28 03:03:34 dist-test-slave-k5rg krb5kdc[1934](info): commencing operation
krb5kdc: starting...
W20260928 03:03:35.289956 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:35 dist-test-slave-k5rg krb5kdc[1934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:35.528168 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:35.843173 1947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:35.843458 1947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:35.843616 1947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:35.850824 1947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:35.850943 1947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:35.851019 1947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:35.851101 1947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:35.851161 1947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:35.860862 1947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:35.862118 1947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:35.863376 1947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:35.874292 1955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:35.874369 1952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:35.874356 1953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:35.875303 1947 server_base.cc:1034] running on GCE node
I20260928 03:03:35.876014 1947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:35.877357 1947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:35.878567 1947 hybrid_clock.cc:648] HybridClock initialized: now 1790564615878516 us; error 40 us; skew 500 ppm
I20260928 03:03:35.878909 1947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:35 dist-test-slave-k5rg krb5kdc[1934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:35.885974 1947 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:35.887346 1947 webserver.cc:460] Webserver started at http://127.0.0.1:40159/ using document root <none> and password file <none>
I20260928 03:03:35.887998 1947 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:35.888118 1947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:35.888401 1947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:35.891320 1947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "752a85ab181645ef92a6d2ec80f0d096"
format_stamp: "Formatted at 2026-09-28 03:03:35 on dist-test-slave-k5rg"
server_key: "190d8419cd908bb27f93600796b57443"
server_key_iv: "1cdc0712374ff6ead908cf7f2d2806ae"
server_key_version: "encryptionkey@0"
I20260928 03:03:35.892050 1947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "752a85ab181645ef92a6d2ec80f0d096"
format_stamp: "Formatted at 2026-09-28 03:03:35 on dist-test-slave-k5rg"
server_key: "190d8419cd908bb27f93600796b57443"
server_key_iv: "1cdc0712374ff6ead908cf7f2d2806ae"
server_key_version: "encryptionkey@0"
I20260928 03:03:35.896862 1947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 03:03:35.900192 1962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:35.901396 1947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 03:03:35.901547 1947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "752a85ab181645ef92a6d2ec80f0d096"
format_stamp: "Formatted at 2026-09-28 03:03:35 on dist-test-slave-k5rg"
server_key: "190d8419cd908bb27f93600796b57443"
server_key_iv: "1cdc0712374ff6ead908cf7f2d2806ae"
server_key_version: "encryptionkey@0"
I20260928 03:03:35.901790 1947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:35.942898 1947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:35.947039 1947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:35.947315 1947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:35.963932 1947 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39739
I20260928 03:03:35.963966 2013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39739 every 8 connection(s)
I20260928 03:03:35.965173 1947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:35.968690 2014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:35.971040 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 1947
I20260928 03:03:35.971194 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:35.971628 13017 external_mini_cluster.cc:1428] Setting key 3327ae33e7baa19855b94a2dbc9f5e69
I20260928 03:03:35.979737 2014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096: Bootstrap starting.
I20260928 03:03:35.983351 2014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:35.984623 2014 log.cc:824] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:03:35 dist-test-slave-k5rg krb5kdc[1934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 03:03:35.987499 2014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096: No bootstrap required, opened a new log
I20260928 03:03:35.993327 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:35.973203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:55979 (local address 127.0.0.1:39739)
0928 03:03:35.973940 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:35.973951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:35.973990 (+ 39us) server_negotiation.cc:407] Connection header received
0928 03:03:35.975604 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:35.975625 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:35.975988 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:35.976303 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:35.978004 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:35.979257 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:35.980912 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:35.981578 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:35.981745 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:35.986505 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:35.986513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:35.986525 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:35.986560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:35.989322 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:35.990235 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:35.990240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:35.990245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:35.990401 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:35.990850 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:35.990853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:35.990855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:35.991335 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:35.991577 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:35.992029 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 03:03:35.992332 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":356,"threads_started":1}
I20260928 03:03:35.996102 2014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752a85ab181645ef92a6d2ec80f0d096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39739 } }
I20260928 03:03:35.996502 2014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:35.996594 2014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752a85ab181645ef92a6d2ec80f0d096, State: Initialized, Role: FOLLOWER
I20260928 03:03:35.997066 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:35.997141 2014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752a85ab181645ef92a6d2ec80f0d096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39739 } }
I20260928 03:03:35.997383 2014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:35.997506 2014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:35.997673 2014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:35.999922 2014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752a85ab181645ef92a6d2ec80f0d096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39739 } }
I20260928 03:03:36.000396 2014 leader_election.cc:302] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 752a85ab181645ef92a6d2ec80f0d096; no voters:
I20260928 03:03:36.000919 2014 leader_election.cc:288] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:36.001053 2023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:36.002292 2023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [term 1 LEADER]: Becoming Leader. State: Replica: 752a85ab181645ef92a6d2ec80f0d096, State: Running, Role: LEADER
I20260928 03:03:36.002835 2023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752a85ab181645ef92a6d2ec80f0d096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39739 } }
I20260928 03:03:36.003181 2014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:36.005330 2024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "752a85ab181645ef92a6d2ec80f0d096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752a85ab181645ef92a6d2ec80f0d096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39739 } } }
I20260928 03:03:36.005703 2024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:36.006176 2025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752a85ab181645ef92a6d2ec80f0d096. Latest consensus state: current_term: 1 leader_uuid: "752a85ab181645ef92a6d2ec80f0d096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752a85ab181645ef92a6d2ec80f0d096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39739 } } }
I20260928 03:03:36.006536 2025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:36.006588 2029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:36.022441 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:36.001955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:58085 (local address 127.0.0.1:39739)
0928 03:03:36.002129 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:36.002136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:36.004830 (+ 2694us) server_negotiation.cc:407] Connection header received
0928 03:03:36.005038 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:36.005044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:36.005153 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:36.005257 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:36.007080 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:36.008294 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:36.010629 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:36.011371 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:36.011470 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:36.014967 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:36.014974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:36.014980 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:36.015035 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:36.018799 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:36.019753 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:36.019758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:36.019761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:36.019905 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:36.020472 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:36.020477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:36.020480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:36.020893 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:36.021170 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:36.021867 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 03:03:36.022064 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 03:03:36.023584 2029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:36.040040 2029 catalog_manager.cc:1347] Generated new cluster ID: 5e3fd3ce7a364f40a1c76895914918df
I20260928 03:03:36.040166 2029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:36.049396 2029 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:36.050592 2029 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:36.059873 2029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 752a85ab181645ef92a6d2ec80f0d096: Generated new TSK 0
I20260928 03:03:36.060658 2029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:36.071427 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 1947
I20260928 03:03:36.072969 2009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:36.162165 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 1947
W20260928 03:03:36.175850 1932 connection.cc:535] client connection to 127.0.0.1:39739 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39739): Connection reset by peer (error 104)
2026-09-28T03:03:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:36 dist-test-slave-k5rg krb5kdc[2046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:36 dist-test-slave-k5rg krb5kdc[2046](info): set up 2 sockets
Sep 28 03:03:36 dist-test-slave-k5rg krb5kdc[2046](info): commencing operation
krb5kdc: starting...
W20260928 03:03:37.181919 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:37 dist-test-slave-k5rg krb5kdc[2046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:37.404672 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:37.715283 2059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:37.715538 2059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:37.715646 2059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:37.722600 2059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:37.722695 2059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:37.722747 2059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:37.722807 2059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:37.722848 2059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:37.732244 2059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:37.733379 2059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:37.734568 2059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:37.745373 2064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:37.745425 2067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:37.745342 2065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:37.746034 2059 server_base.cc:1034] running on GCE node
I20260928 03:03:37.746861 2059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:37.748170 2059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:37.749382 2059 hybrid_clock.cc:648] HybridClock initialized: now 1790564617749336 us; error 40 us; skew 500 ppm
I20260928 03:03:37.749748 2059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:37 dist-test-slave-k5rg krb5kdc[2046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:37.752606 2059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:37.754020 2059 webserver.cc:460] Webserver started at http://127.0.0.1:36509/ using document root <none> and password file <none>
I20260928 03:03:37.754647 2059 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:37.754760 2059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:37.755024 2059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:37.757884 2059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "3a685622ced549fead313bbb3f0e8db6"
format_stamp: "Formatted at 2026-09-28 03:03:37 on dist-test-slave-k5rg"
server_key: "1bcfa684e118740961d67575afd8ab8d"
server_key_iv: "1162d57ab7a0b10655236b7edb9eb33b"
server_key_version: "encryptionkey@0"
I20260928 03:03:37.758580 2059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "3a685622ced549fead313bbb3f0e8db6"
format_stamp: "Formatted at 2026-09-28 03:03:37 on dist-test-slave-k5rg"
server_key: "1bcfa684e118740961d67575afd8ab8d"
server_key_iv: "1162d57ab7a0b10655236b7edb9eb33b"
server_key_version: "encryptionkey@0"
I20260928 03:03:37.763643 2059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:03:37.766878 2074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:37.768116 2059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:03:37.768278 2059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "3a685622ced549fead313bbb3f0e8db6"
format_stamp: "Formatted at 2026-09-28 03:03:37 on dist-test-slave-k5rg"
server_key: "1bcfa684e118740961d67575afd8ab8d"
server_key_iv: "1162d57ab7a0b10655236b7edb9eb33b"
server_key_version: "encryptionkey@0"
I20260928 03:03:37.768447 2059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:37.791236 2059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:37.794945 2059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:37.795197 2059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:37.811729 2059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45979
I20260928 03:03:37.811750 2125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45979 every 8 connection(s)
I20260928 03:03:37.813019 2059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:37.814814 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 2059
I20260928 03:03:37.814975 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:37.815416 13017 external_mini_cluster.cc:1428] Setting key 31e58caecb325e234bfc5f5f85f281a7
I20260928 03:03:37.816902 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:37.829572 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:37.817093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:37631 (local address 127.0.0.1:45979)
0928 03:03:37.817844 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:37.817859 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:37.817907 (+ 48us) server_negotiation.cc:407] Connection header received
0928 03:03:37.819896 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:37.819923 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:37.820433 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:37.820847 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:37.821345 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:37.823317 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:37.824613 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:37.825583 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:37.825779 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:37.826892 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:37.826902 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:37.826920 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:37.827635 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:37.827728 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:37.827763 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:37.827915 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:37.828183 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 03:03:37.828360 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":283,"threads_started":1}
I20260928 03:03:37.830863 2126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6: Bootstrap starting.
I20260928 03:03:37.833194 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:37.834090 2126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:37.835276 2126 log.cc:824] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:37.838179 2126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6: No bootstrap required, opened a new log
I20260928 03:03:37.844542 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:37.838157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:60107 (local address 127.0.0.1:45979)
0928 03:03:37.838342 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:37.838347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:37.838667 (+ 320us) server_negotiation.cc:407] Connection header received
0928 03:03:37.838843 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:37.838847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:37.838907 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:37.838987 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:37.839583 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:37.840364 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:37.842208 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:37.842639 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:37.842731 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:37.843840 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:37.843844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:37.843847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:37.843865 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:37.843902 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:37.843916 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:37.843995 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:37.844180 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 03:03:37.844228 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:03:37.847188 2126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a685622ced549fead313bbb3f0e8db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } }
I20260928 03:03:37.847548 2126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:37.847637 2126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a685622ced549fead313bbb3f0e8db6, State: Initialized, Role: FOLLOWER
I20260928 03:03:37.848150 2126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a685622ced549fead313bbb3f0e8db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } }
I20260928 03:03:37.848356 2126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:37.848474 2126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:37.848631 2126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:37.850843 2126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a685622ced549fead313bbb3f0e8db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } }
I20260928 03:03:37.851297 2126 leader_election.cc:302] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a685622ced549fead313bbb3f0e8db6; no voters:
I20260928 03:03:37.851822 2126 leader_election.cc:288] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:37.851930 2136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:37.852535 2136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [term 1 LEADER]: Becoming Leader. State: Replica: 3a685622ced549fead313bbb3f0e8db6, State: Running, Role: LEADER
I20260928 03:03:37.853039 2136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a685622ced549fead313bbb3f0e8db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } }
I20260928 03:03:37.853724 2126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:37.855778 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a685622ced549fead313bbb3f0e8db6. Latest consensus state: current_term: 1 leader_uuid: "3a685622ced549fead313bbb3f0e8db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a685622ced549fead313bbb3f0e8db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } } }
I20260928 03:03:37.855767 2137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a685622ced549fead313bbb3f0e8db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a685622ced549fead313bbb3f0e8db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } } }
I20260928 03:03:37.856181 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:37.856190 2137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:37.866156 2144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:37.871110 2144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:37.879791 2144 catalog_manager.cc:1347] Generated new cluster ID: 3a9347d8652b48829dbbc27d074037bb
I20260928 03:03:37.879881 2144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:37.934178 2144 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:37.935552 2144 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:37.944407 2144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a685622ced549fead313bbb3f0e8db6: Generated new TSK 0
I20260928 03:03:37.945187 2144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:37.957623 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2059
W20260928 03:03:37.958930 2121 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:37631
W20260928 03:03:37.959650 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2059: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:37.959782 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 2059
W20260928 03:03:37.973380 2044 connection.cc:535] client connection to 127.0.0.1:45979 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45979): Connection reset by peer (error 104)
2026-09-28T03:03:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1797 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:38 dist-test-slave-k5rg krb5kdc[2157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:38 dist-test-slave-k5rg krb5kdc[2157](info): set up 2 sockets
Sep 28 03:03:38 dist-test-slave-k5rg krb5kdc[2157](info): commencing operation
krb5kdc: starting...
W20260928 03:03:38.969913 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:39 dist-test-slave-k5rg krb5kdc[2157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:03:39.204474 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:39.513996 2170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:39.514277 2170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:39.514413 2170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:39.521382 2170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:39.521495 2170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:39.521576 2170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:39.521675 2170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:39.521739 2170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:39.531250 2170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:39.532462 2170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:39.533694 2170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:39.544452 2178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:39.544485 2176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:39.544513 2175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:39.545408 2170 server_base.cc:1034] running on GCE node
I20260928 03:03:39.546159 2170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:39.547389 2170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:39.548574 2170 hybrid_clock.cc:648] HybridClock initialized: now 1790564619548528 us; error 34 us; skew 500 ppm
I20260928 03:03:39.548880 2170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:39 dist-test-slave-k5rg krb5kdc[2157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:39.551671 2170 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:03:39.553013 2170 webserver.cc:460] Webserver started at http://127.0.0.1:44549/ using document root <none> and password file <none>
I20260928 03:03:39.553637 2170 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:39.553745 2170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:39.553988 2170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:39.556733 2170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "50dd38ee04a44319b5b908e1f9693934"
format_stamp: "Formatted at 2026-09-28 03:03:39 on dist-test-slave-k5rg"
server_key: "4d1be70f4eeaa7870b6b28504e048c34"
server_key_iv: "dddf7ad66c0a2045bf975641f7c70059"
server_key_version: "encryptionkey@0"
I20260928 03:03:39.557407 2170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "50dd38ee04a44319b5b908e1f9693934"
format_stamp: "Formatted at 2026-09-28 03:03:39 on dist-test-slave-k5rg"
server_key: "4d1be70f4eeaa7870b6b28504e048c34"
server_key_iv: "dddf7ad66c0a2045bf975641f7c70059"
server_key_version: "encryptionkey@0"
I20260928 03:03:39.562069 2170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:03:39.565201 2185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:39.566448 2170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:03:39.566593 2170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "50dd38ee04a44319b5b908e1f9693934"
format_stamp: "Formatted at 2026-09-28 03:03:39 on dist-test-slave-k5rg"
server_key: "4d1be70f4eeaa7870b6b28504e048c34"
server_key_iv: "dddf7ad66c0a2045bf975641f7c70059"
server_key_version: "encryptionkey@0"
I20260928 03:03:39.566762 2170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:39.572157 2170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:39.575768 2170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:39.576021 2170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:39.592399 2170 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40497
I20260928 03:03:39.592439 2236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40497 every 8 connection(s)
I20260928 03:03:39.593699 2170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:39.595278 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 2170
I20260928 03:03:39.595459 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:39.595980 13017 external_mini_cluster.cc:1428] Setting key 6731cd2564c08dad2141027a642ea61e
I20260928 03:03:39.597564 2237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:39.603099 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:39.597736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:46793 (local address 127.0.0.1:40497)
0928 03:03:39.598513 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:39.598523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:39.598560 (+ 37us) server_negotiation.cc:407] Connection header received
0928 03:03:39.600040 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:39.600060 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:39.600395 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:39.600689 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:39.601674 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:39.601681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:39.601693 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:39.601710 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:39.601790 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:39.601813 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:39.601972 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:39.602065 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 03:03:39.602248 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":342,"threads_started":1}
I20260928 03:03:39.606359 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:39.609925 2237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934: Bootstrap starting.
I20260928 03:03:39.613507 2237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:39.613548 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:39.611427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:53169 (local address 127.0.0.1:40497)
0928 03:03:39.611594 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:39.611599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:39.611857 (+ 258us) server_negotiation.cc:407] Connection header received
0928 03:03:39.612038 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:39.612042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:39.612082 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:39.612148 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:39.612997 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:39.613001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:39.613004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:39.613016 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:39.613050 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:39.613064 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:39.613139 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:39.613260 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 03:03:39.613301 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 03:03:39.614547 2237 log.cc:824] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:39.616858 2237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934: No bootstrap required, opened a new log
I20260928 03:03:39.623603 2237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50dd38ee04a44319b5b908e1f9693934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260928 03:03:39.623937 2237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:39.624022 2237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50dd38ee04a44319b5b908e1f9693934, State: Initialized, Role: FOLLOWER
I20260928 03:03:39.624595 2237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50dd38ee04a44319b5b908e1f9693934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260928 03:03:39.624842 2237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:39.624969 2237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:39.625118 2237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:39.627166 2237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50dd38ee04a44319b5b908e1f9693934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260928 03:03:39.627597 2237 leader_election.cc:302] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50dd38ee04a44319b5b908e1f9693934; no voters:
I20260928 03:03:39.628150 2237 leader_election.cc:288] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:39.628288 2247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:39.629055 2247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [term 1 LEADER]: Becoming Leader. State: Replica: 50dd38ee04a44319b5b908e1f9693934, State: Running, Role: LEADER
I20260928 03:03:39.629544 2247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50dd38ee04a44319b5b908e1f9693934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260928 03:03:39.630056 2237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:39.632081 2249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50dd38ee04a44319b5b908e1f9693934. Latest consensus state: current_term: 1 leader_uuid: "50dd38ee04a44319b5b908e1f9693934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50dd38ee04a44319b5b908e1f9693934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260928 03:03:39.632092 2248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50dd38ee04a44319b5b908e1f9693934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50dd38ee04a44319b5b908e1f9693934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260928 03:03:39.632535 2249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:39.632586 2248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:39.634408 2252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:39.638432 2252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:39.645545 2252 catalog_manager.cc:1347] Generated new cluster ID: 17bae4778a9b46b6b6f6f16c472d7d0a
I20260928 03:03:39.645658 2252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:39.670722 2252 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:39.671208 2252 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:39.683310 2252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50dd38ee04a44319b5b908e1f9693934: Generated new TSK 0
I20260928 03:03:39.684340 2252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:39.693380 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2170
W20260928 03:03:39.694442 2232 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:46793
W20260928 03:03:39.695042 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2170: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:39.695169 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 2170
2026-09-28T03:03:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1735 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:39 dist-test-slave-k5rg krb5kdc[2268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:39 dist-test-slave-k5rg krb5kdc[2268](info): set up 2 sockets
Sep 28 03:03:39 dist-test-slave-k5rg krb5kdc[2268](info): commencing operation
krb5kdc: starting...
W20260928 03:03:40.701938 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:40 dist-test-slave-k5rg krb5kdc[2268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:03:40.940244 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:38961
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:38961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:41.257778 2281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:41.258065 2281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:41.258200 2281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:41.268118 2281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:41.268265 2281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:41.268347 2281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:41.268432 2281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:41.268503 2281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:41.283964 2281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:38961
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:38961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:41.285771 2281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:41.287585 2281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:41.299697 2289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:41.299757 2286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:41.299922 2287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:41.305855 2281 server_base.cc:1034] running on GCE node
I20260928 03:03:41.306658 2281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:41.308074 2281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:41.309291 2281 hybrid_clock.cc:648] HybridClock initialized: now 1790564621309228 us; error 60 us; skew 500 ppm
I20260928 03:03:41.309690 2281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:41 dist-test-slave-k5rg krb5kdc[2268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564621, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:41.312839 2281 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:03:41.314304 2281 webserver.cc:460] Webserver started at http://127.12.182.126:38387/ using document root <none> and password file <none>
I20260928 03:03:41.314961 2281 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:41.315080 2281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:41.315393 2281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:41.318279 2281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "69c669603b644b7daa28b8c211b71847"
format_stamp: "Formatted at 2026-09-28 03:03:41 on dist-test-slave-k5rg"
server_key: "bf5238b142545ec93e5f6d2201f716d6"
server_key_iv: "9f240245c25952f1e31beb7c26bc2449"
server_key_version: "encryptionkey@0"
I20260928 03:03:41.318984 2281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "69c669603b644b7daa28b8c211b71847"
format_stamp: "Formatted at 2026-09-28 03:03:41 on dist-test-slave-k5rg"
server_key: "bf5238b142545ec93e5f6d2201f716d6"
server_key_iv: "9f240245c25952f1e31beb7c26bc2449"
server_key_version: "encryptionkey@0"
I20260928 03:03:41.323812 2281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 03:03:41.326980 2296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:41.328179 2281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 03:03:41.328334 2281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "69c669603b644b7daa28b8c211b71847"
format_stamp: "Formatted at 2026-09-28 03:03:41 on dist-test-slave-k5rg"
server_key: "bf5238b142545ec93e5f6d2201f716d6"
server_key_iv: "9f240245c25952f1e31beb7c26bc2449"
server_key_version: "encryptionkey@0"
I20260928 03:03:41.328496 2281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:41.361195 2281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:41.365316 2281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:41.365576 2281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:41.382445 2281 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:38961
I20260928 03:03:41.382468 2347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:38961 every 8 connection(s)
I20260928 03:03:41.383752 2281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:41.387269 2348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:41.392732 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 2281
I20260928 03:03:41.392901 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:41.393287 13017 external_mini_cluster.cc:1428] Setting key 9578129b687e74e3147547082bdd3cfc
I20260928 03:03:41.397743 2348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847: Bootstrap starting.
I20260928 03:03:41.401399 2348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:41.402674 2348 log.cc:824] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:41.405966 2348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847: No bootstrap required, opened a new log
Sep 28 03:03:41 dist-test-slave-k5rg krb5kdc[2268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:41.416410 2348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c669603b644b7daa28b8c211b71847" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38961 } }
I20260928 03:03:41.416934 2348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:41.417047 2348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c669603b644b7daa28b8c211b71847, State: Initialized, Role: FOLLOWER
I20260928 03:03:41.417855 2348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c669603b644b7daa28b8c211b71847" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38961 } }
I20260928 03:03:41.418141 2348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:41.418279 2348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:41.418481 2348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:41.421691 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:41.395098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53799 (local address 127.12.182.126:38961)
0928 03:03:41.396007 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:41.396022 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:41.396349 (+ 327us) server_negotiation.cc:407] Connection header received
0928 03:03:41.398721 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:41.398747 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:41.399245 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:41.399688 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:41.401024 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:41.403008 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:41.404562 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:41.405550 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:41.405815 (+ 265us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:41.411521 (+ 5706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:41.411531 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:41.411551 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:41.411597 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:41.415721 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:41.416779 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:41.416788 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:41.416796 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:41.417011 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:41.417655 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:41.417660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:41.417663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:41.418388 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:41.418703 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:41.419944 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0928 03:03:41.420308 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":408,"threads_started":1}
I20260928 03:03:41.421648 2348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c669603b644b7daa28b8c211b71847" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38961 } }
I20260928 03:03:41.422204 2348 leader_election.cc:302] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c669603b644b7daa28b8c211b71847; no voters:
I20260928 03:03:41.422741 2348 leader_election.cc:288] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:41.422883 2353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:41.423527 2353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [term 1 LEADER]: Becoming Leader. State: Replica: 69c669603b644b7daa28b8c211b71847, State: Running, Role: LEADER
I20260928 03:03:41.424149 2353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c669603b644b7daa28b8c211b71847" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38961 } }
I20260928 03:03:41.426011 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:41.426066 2348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:41.432127 2354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c669603b644b7daa28b8c211b71847. Latest consensus state: current_term: 1 leader_uuid: "69c669603b644b7daa28b8c211b71847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c669603b644b7daa28b8c211b71847" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38961 } } }
I20260928 03:03:41.432524 2354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:41.433486 2367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:41.433351 2355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c669603b644b7daa28b8c211b71847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c669603b644b7daa28b8c211b71847" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 38961 } } }
I20260928 03:03:41.433760 2355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:41.438968 2367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:41.449651 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:41.431871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42491 (local address 127.12.182.126:38961)
0928 03:03:41.432036 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:41.432041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:41.432060 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:03:41.432375 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:41.432381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:41.432446 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:41.432528 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:41.434481 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:41.435621 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:41.437380 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:41.438055 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:41.438165 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:41.441540 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:41.441547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:41.441552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:41.441596 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:41.445816 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:41.446876 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:41.446882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:41.446885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:41.447047 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:41.447631 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:41.447637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:41.447640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:41.448076 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:41.448337 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:41.449020 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 03:03:41.449233 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 03:03:41.454699 2367 catalog_manager.cc:1347] Generated new cluster ID: 88cfa370a9504100a05dd9b9d5c712a5
I20260928 03:03:41.454806 2367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:41.465732 2367 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:41.467218 2367 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:41.473351 2367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c669603b644b7daa28b8c211b71847: Generated new TSK 0
I20260928 03:03:41.474426 2367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:41.483528 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2281
I20260928 03:03:41.485044 2343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:41.573787 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 2281
W20260928 03:03:41.587430 2266 connection.cc:535] client connection to 127.12.182.126:38961 recv error: Network error: failed to read from TLS socket (remote: 127.12.182.126:38961): Connection reset by peer (error 104)
2026-09-28T03:03:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:41 dist-test-slave-k5rg krb5kdc[2380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:41 dist-test-slave-k5rg krb5kdc[2380](info): set up 2 sockets
Sep 28 03:03:41 dist-test-slave-k5rg krb5kdc[2380](info): commencing operation
krb5kdc: starting...
W20260928 03:03:42.585991 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:42 dist-test-slave-k5rg krb5kdc[2380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:03:42.816689 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:43589
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:43589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:43.138445 2393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:43.138720 2393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:43.138829 2393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:43.145877 2393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:43.145985 2393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:43.146029 2393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:43.146090 2393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:43.146130 2393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:43.155593 2393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:43589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:43589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:43.156730 2393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:43.158002 2393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:43.169214 2398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:43.169098 2399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:43.169157 2401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:43.169900 2393 server_base.cc:1034] running on GCE node
I20260928 03:03:43.170600 2393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:43.171919 2393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:43.173134 2393 hybrid_clock.cc:648] HybridClock initialized: now 1790564623173072 us; error 51 us; skew 500 ppm
I20260928 03:03:43.173468 2393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:43 dist-test-slave-k5rg krb5kdc[2380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564623, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:43.176321 2393 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:03:43.177729 2393 webserver.cc:460] Webserver started at http://127.12.182.126:39581/ using document root <none> and password file <none>
I20260928 03:03:43.178376 2393 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:43.178491 2393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:43.178771 2393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:43.181567 2393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "fec35572ba644c86913e84aa18ffb301"
format_stamp: "Formatted at 2026-09-28 03:03:43 on dist-test-slave-k5rg"
server_key: "6ed1d772e0bf1d0fb94c60ae36b54f01"
server_key_iv: "dea428d50113a5a1fd24bcdacbaf2d4b"
server_key_version: "encryptionkey@0"
I20260928 03:03:43.182337 2393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "fec35572ba644c86913e84aa18ffb301"
format_stamp: "Formatted at 2026-09-28 03:03:43 on dist-test-slave-k5rg"
server_key: "6ed1d772e0bf1d0fb94c60ae36b54f01"
server_key_iv: "dea428d50113a5a1fd24bcdacbaf2d4b"
server_key_version: "encryptionkey@0"
I20260928 03:03:43.187130 2393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:03:43.190452 2408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:43.191694 2393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:03:43.191831 2393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "fec35572ba644c86913e84aa18ffb301"
format_stamp: "Formatted at 2026-09-28 03:03:43 on dist-test-slave-k5rg"
server_key: "6ed1d772e0bf1d0fb94c60ae36b54f01"
server_key_iv: "dea428d50113a5a1fd24bcdacbaf2d4b"
server_key_version: "encryptionkey@0"
I20260928 03:03:43.191995 2393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:43.205523 2393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:43.209312 2393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:43.209539 2393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:43.226713 2393 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:43589
I20260928 03:03:43.226737 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:43589 every 8 connection(s)
I20260928 03:03:43.227962 2393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:43.228917 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 2393
I20260928 03:03:43.229108 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:43.229682 13017 external_mini_cluster.cc:1428] Setting key 44fbfd58ca95372593664a841c9f652b
I20260928 03:03:43.231879 2460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:43.245769 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:43.231479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38729 (local address 127.12.182.126:43589)
0928 03:03:43.232312 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:43.232325 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:43.234127 (+ 1802us) server_negotiation.cc:407] Connection header received
0928 03:03:43.236153 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:43.236179 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:43.236665 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:43.237090 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:43.237592 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:43.239579 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:43.240978 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:43.241985 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:43.242159 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:43.243199 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:43.243212 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:43.243230 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:43.243948 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:43.244035 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:43.244075 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:43.244226 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:43.244405 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 03:03:43.244566 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":363,"threads_started":1}
I20260928 03:03:43.245954 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301: Bootstrap starting.
I20260928 03:03:43.248905 2460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:43.249353 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:43.250033 2460 log.cc:824] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:43.253014 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301: No bootstrap required, opened a new log
I20260928 03:03:43.260222 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:43.253852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51915 (local address 127.12.182.126:43589)
0928 03:03:43.254011 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:43.254017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:43.254475 (+ 458us) server_negotiation.cc:407] Connection header received
0928 03:03:43.254611 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:43.254616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:43.254688 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:43.254798 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:43.255226 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:43.256374 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:43.257668 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:43.258381 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:43.258507 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:43.259462 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:43.259469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:43.259473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:43.259492 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:43.259549 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:43.259568 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:43.259666 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:43.259833 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 03:03:43.259888 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 03:03:43.262110 2460 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec35572ba644c86913e84aa18ffb301" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43589 } }
I20260928 03:03:43.262449 2460 raft_consensus.cc:374] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:43.262544 2460 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec35572ba644c86913e84aa18ffb301, State: Initialized, Role: FOLLOWER
I20260928 03:03:43.263101 2460 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec35572ba644c86913e84aa18ffb301" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43589 } }
I20260928 03:03:43.263304 2460 raft_consensus.cc:388] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:43.263434 2460 raft_consensus.cc:482] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:43.263592 2460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:43.265738 2460 raft_consensus.cc:504] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec35572ba644c86913e84aa18ffb301" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43589 } }
I20260928 03:03:43.266211 2460 leader_election.cc:302] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fec35572ba644c86913e84aa18ffb301; no voters:
I20260928 03:03:43.266769 2460 leader_election.cc:288] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:43.266908 2470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:43.267884 2470 raft_consensus.cc:686] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [term 1 LEADER]: Becoming Leader. State: Replica: fec35572ba644c86913e84aa18ffb301, State: Running, Role: LEADER
I20260928 03:03:43.268482 2470 consensus_queue.cc:237] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec35572ba644c86913e84aa18ffb301" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43589 } }
I20260928 03:03:43.268746 2460 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:43.271071 2472 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fec35572ba644c86913e84aa18ffb301. Latest consensus state: current_term: 1 leader_uuid: "fec35572ba644c86913e84aa18ffb301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec35572ba644c86913e84aa18ffb301" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43589 } } }
I20260928 03:03:43.271466 2472 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:43.272006 2471 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fec35572ba644c86913e84aa18ffb301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec35572ba644c86913e84aa18ffb301" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 43589 } } }
I20260928 03:03:43.272481 2471 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:43.275979 2480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:43.280210 2480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:43.287999 2480 catalog_manager.cc:1347] Generated new cluster ID: f98f73fa499a4ac39360b8459ed35537
I20260928 03:03:43.288105 2480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:43.315791 2480 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:43.317581 2480 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:43.326841 2480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fec35572ba644c86913e84aa18ffb301: Generated new TSK 0
I20260928 03:03:43.327553 2480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:43.344216 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2393
W20260928 03:03:43.345741 2455 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38729
W20260928 03:03:43.346379 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2393: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:43.346504 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 2393
W20260928 03:03:43.360204 2378 connection.cc:535] client connection to 127.12.182.126:43589 recv error: Network error: failed to read from TLS socket (remote: 127.12.182.126:43589): Connection reset by peer (error 104)
2026-09-28T03:03:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1768 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:43 dist-test-slave-k5rg krb5kdc[2492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:43 dist-test-slave-k5rg krb5kdc[2492](info): set up 2 sockets
Sep 28 03:03:43 dist-test-slave-k5rg krb5kdc[2492](info): commencing operation
krb5kdc: starting...
W20260928 03:03:44.353943 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:44 dist-test-slave-k5rg krb5kdc[2492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:03:44.597481 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:42619
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:42619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:44.914537 2505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:44.914815 2505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:44.914947 2505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:44.922200 2505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:44.922318 2505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:44.922392 2505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:44.922468 2505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:44.922524 2505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:44.932294 2505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:42619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:42619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:44.933552 2505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:44.934854 2505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:44.945815 2510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:44.945755 2513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:44.945783 2511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:44.946581 2505 server_base.cc:1034] running on GCE node
I20260928 03:03:44.947337 2505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:44.948640 2505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:44.949846 2505 hybrid_clock.cc:648] HybridClock initialized: now 1790564624949793 us; error 40 us; skew 500 ppm
I20260928 03:03:44.950176 2505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:44 dist-test-slave-k5rg krb5kdc[2492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:44.953025 2505 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:03:44.954444 2505 webserver.cc:460] Webserver started at http://127.12.182.126:40599/ using document root <none> and password file <none>
I20260928 03:03:44.955102 2505 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:44.955216 2505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:44.955490 2505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:44.958386 2505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "c5325462f69543f18cf209b484a146ae"
format_stamp: "Formatted at 2026-09-28 03:03:44 on dist-test-slave-k5rg"
server_key: "ed521b867464ec341375c3818326dcd5"
server_key_iv: "6e6c1291465a1aad0cfbf0a8e912e3b0"
server_key_version: "encryptionkey@0"
I20260928 03:03:44.959120 2505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "c5325462f69543f18cf209b484a146ae"
format_stamp: "Formatted at 2026-09-28 03:03:44 on dist-test-slave-k5rg"
server_key: "ed521b867464ec341375c3818326dcd5"
server_key_iv: "6e6c1291465a1aad0cfbf0a8e912e3b0"
server_key_version: "encryptionkey@0"
I20260928 03:03:44.963979 2505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:03:44.967238 2520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:44.968453 2505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:03:44.968606 2505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "c5325462f69543f18cf209b484a146ae"
format_stamp: "Formatted at 2026-09-28 03:03:44 on dist-test-slave-k5rg"
server_key: "ed521b867464ec341375c3818326dcd5"
server_key_iv: "6e6c1291465a1aad0cfbf0a8e912e3b0"
server_key_version: "encryptionkey@0"
I20260928 03:03:44.968799 2505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:44.974260 2505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:44.977895 2505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:44.978149 2505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:44.994848 2505 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:42619
I20260928 03:03:44.994828 2571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:42619 every 8 connection(s)
I20260928 03:03:44.996174 2505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:44.999709 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 2505
I20260928 03:03:44.999859 2572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:44.999892 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:45.000432 13017 external_mini_cluster.cc:1428] Setting key c77831ac5e4ec61e395fe9aba90cf6ff
I20260928 03:03:45.007686 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:45.002133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40959 (local address 127.12.182.126:42619)
0928 03:03:45.002957 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:45.002971 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:45.003010 (+ 39us) server_negotiation.cc:407] Connection header received
0928 03:03:45.004627 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:45.004649 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:45.004982 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:45.005306 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:45.006232 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:45.006238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:45.006250 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:45.006266 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:45.006329 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:45.006348 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:45.006519 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:45.006609 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 03:03:45.006778 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":381,"threads_started":1}
I20260928 03:03:45.011087 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:45.013290 2572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae: Bootstrap starting.
I20260928 03:03:45.017249 2572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:45.018491 2572 log.cc:824] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:45.018718 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:45.016268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33115 (local address 127.12.182.126:42619)
0928 03:03:45.016456 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:45.016463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:45.016834 (+ 371us) server_negotiation.cc:407] Connection header received
0928 03:03:45.016976 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:45.016981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:45.017044 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:45.017183 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:45.018083 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:45.018088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:45.018092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:45.018110 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:45.018172 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:45.018194 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:45.018317 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:45.018361 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 03:03:45.018415 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:03:45.021251 2572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae: No bootstrap required, opened a new log
I20260928 03:03:45.028327 2572 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5325462f69543f18cf209b484a146ae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42619 } }
I20260928 03:03:45.028651 2572 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:45.028723 2572 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5325462f69543f18cf209b484a146ae, State: Initialized, Role: FOLLOWER
I20260928 03:03:45.029183 2572 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5325462f69543f18cf209b484a146ae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42619 } }
I20260928 03:03:45.029392 2572 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:45.029521 2572 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:45.029745 2572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:45.031904 2572 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5325462f69543f18cf209b484a146ae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42619 } }
I20260928 03:03:45.032371 2572 leader_election.cc:302] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5325462f69543f18cf209b484a146ae; no voters:
I20260928 03:03:45.032922 2572 leader_election.cc:288] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:45.033048 2582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:45.033696 2582 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [term 1 LEADER]: Becoming Leader. State: Replica: c5325462f69543f18cf209b484a146ae, State: Running, Role: LEADER
I20260928 03:03:45.034284 2582 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5325462f69543f18cf209b484a146ae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42619 } }
I20260928 03:03:45.035065 2572 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:45.036831 2583 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5325462f69543f18cf209b484a146ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5325462f69543f18cf209b484a146ae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42619 } } }
I20260928 03:03:45.037129 2583 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:45.038529 2587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:45.042104 2584 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5325462f69543f18cf209b484a146ae. Latest consensus state: current_term: 1 leader_uuid: "c5325462f69543f18cf209b484a146ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5325462f69543f18cf209b484a146ae" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 42619 } } }
I20260928 03:03:45.042521 2584 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:45.043387 2587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:45.052709 2587 catalog_manager.cc:1347] Generated new cluster ID: 069dc392cb4f41988e523f52a1fde4b1
I20260928 03:03:45.052860 2587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:45.071213 2587 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:45.071717 2587 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:45.083390 2587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5325462f69543f18cf209b484a146ae: Generated new TSK 0
I20260928 03:03:45.084519 2587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:45.100169 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2505
W20260928 03:03:45.101282 2567 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40959
W20260928 03:03:45.102054 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2505: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:45.102211 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 2505
2026-09-28T03:03:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1755 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:45 dist-test-slave-k5rg krb5kdc[2603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:45 dist-test-slave-k5rg krb5kdc[2603](info): set up 2 sockets
Sep 28 03:03:45 dist-test-slave-k5rg krb5kdc[2603](info): commencing operation
krb5kdc: starting...
W20260928 03:03:46.114049 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:46 dist-test-slave-k5rg krb5kdc[2603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:03:46.370666 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:35193
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:44395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:35193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:46.699155 2616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:46.699481 2616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:46.699615 2616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:46.706881 2616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:46.707001 2616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:46.707079 2616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:46.707165 2616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:46.707223 2616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:46.716887 2616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:44395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:35193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:35193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:46.718207 2616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:46.719523 2616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:46.730643 2621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:46.730666 2624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:46.730754 2622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:46.731338 2616 server_base.cc:1034] running on GCE node
I20260928 03:03:46.732051 2616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:46.733414 2616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:46.734601 2616 hybrid_clock.cc:648] HybridClock initialized: now 1790564626734544 us; error 51 us; skew 500 ppm
I20260928 03:03:46.734938 2616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:46 dist-test-slave-k5rg krb5kdc[2603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564626, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:46.741822 2616 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:03:46.743309 2616 webserver.cc:460] Webserver started at http://127.12.182.126:34617/ using document root <none> and password file <none>
I20260928 03:03:46.743969 2616 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:46.744078 2616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:46.744351 2616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:46.747295 2616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "c28e931feed147b69728d51cdeed8a76"
format_stamp: "Formatted at 2026-09-28 03:03:46 on dist-test-slave-k5rg"
server_key: "1b74db0e1ecf8294e2102172eb16cd84"
server_key_iv: "87cc9a5e8d7f5f2b7c94c2f3755dca43"
server_key_version: "encryptionkey@0"
I20260928 03:03:46.748008 2616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "c28e931feed147b69728d51cdeed8a76"
format_stamp: "Formatted at 2026-09-28 03:03:46 on dist-test-slave-k5rg"
server_key: "1b74db0e1ecf8294e2102172eb16cd84"
server_key_iv: "87cc9a5e8d7f5f2b7c94c2f3755dca43"
server_key_version: "encryptionkey@0"
I20260928 03:03:46.752919 2616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 03:03:46.756203 2631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:46.757448 2616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:03:46.757640 2616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "c28e931feed147b69728d51cdeed8a76"
format_stamp: "Formatted at 2026-09-28 03:03:46 on dist-test-slave-k5rg"
server_key: "1b74db0e1ecf8294e2102172eb16cd84"
server_key_iv: "87cc9a5e8d7f5f2b7c94c2f3755dca43"
server_key_version: "encryptionkey@0"
I20260928 03:03:46.757825 2616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:46.772996 2616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:46.777030 2616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:46.777289 2616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:46.796499 2616 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:35193
I20260928 03:03:46.796519 2682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:35193 every 8 connection(s)
I20260928 03:03:46.797868 2616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:46.801469 2683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:46.804430 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 2616
I20260928 03:03:46.804618 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:46.805028 13017 external_mini_cluster.cc:1428] Setting key 315ef12434e5a8bec83a0b58c13ce7ae
I20260928 03:03:46.812789 2683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76: Bootstrap starting.
I20260928 03:03:46.816316 2683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:46.817555 2683 log.cc:824] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:03:46 dist-test-slave-k5rg krb5kdc[2603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:46.820602 2683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76: No bootstrap required, opened a new log
I20260928 03:03:46.827062 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:46.806611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38083 (local address 127.12.182.126:35193)
0928 03:03:46.807320 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:46.807331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:46.807370 (+ 39us) server_negotiation.cc:407] Connection header received
0928 03:03:46.808998 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:46.809020 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:46.809408 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:46.809774 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:46.811413 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:46.812643 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:46.814361 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:46.815035 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:46.815154 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:46.819915 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:46.819923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:46.819937 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:46.819974 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:46.822927 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:46.823813 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:46.823819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:46.823828 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:46.823997 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:46.824391 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:46.824394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:46.824396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:46.824924 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:46.825145 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:46.825656 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 03:03:46.826052 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":324,"threads_started":1}
I20260928 03:03:46.828989 2683 raft_consensus.cc:348] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28e931feed147b69728d51cdeed8a76" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35193 } }
I20260928 03:03:46.829340 2683 raft_consensus.cc:374] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:46.829440 2683 raft_consensus.cc:729] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28e931feed147b69728d51cdeed8a76, State: Initialized, Role: FOLLOWER
I20260928 03:03:46.830020 2683 consensus_queue.cc:260] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28e931feed147b69728d51cdeed8a76" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35193 } }
I20260928 03:03:46.830266 2683 raft_consensus.cc:388] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:46.830394 2683 raft_consensus.cc:482] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:46.830451 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:46.830554 2683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:46.832628 2683 raft_consensus.cc:504] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28e931feed147b69728d51cdeed8a76" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35193 } }
I20260928 03:03:46.833088 2683 leader_election.cc:302] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c28e931feed147b69728d51cdeed8a76; no voters:
I20260928 03:03:46.833724 2683 leader_election.cc:288] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:46.833801 2692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:46.835928 2683 sys_catalog.cc:564] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:46.840652 2692 raft_consensus.cc:686] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [term 1 LEADER]: Becoming Leader. State: Replica: c28e931feed147b69728d51cdeed8a76, State: Running, Role: LEADER
I20260928 03:03:46.841310 2692 consensus_queue.cc:237] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28e931feed147b69728d51cdeed8a76" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35193 } }
I20260928 03:03:46.844995 2693 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c28e931feed147b69728d51cdeed8a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28e931feed147b69728d51cdeed8a76" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35193 } } }
I20260928 03:03:46.845397 2693 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:46.844970 2694 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c28e931feed147b69728d51cdeed8a76. Latest consensus state: current_term: 1 leader_uuid: "c28e931feed147b69728d51cdeed8a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c28e931feed147b69728d51cdeed8a76" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 35193 } } }
I20260928 03:03:46.846000 2694 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:46.850203 2702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:46.855485 2702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:46.858284 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:46.834979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37365 (local address 127.12.182.126:35193)
0928 03:03:46.835150 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:46.835156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:46.841401 (+ 6245us) server_negotiation.cc:407] Connection header received
0928 03:03:46.841575 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:46.841581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:46.841681 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:46.841764 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:46.843265 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:46.844344 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:46.846133 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:46.846884 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:46.846984 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:46.850274 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:46.850281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:46.850286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:46.850336 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:46.854326 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:46.855144 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:46.855152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:46.855155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:46.855336 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:46.855780 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:46.855785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:46.855788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:46.856212 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:46.857527 (+ 1315us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:46.857702 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 03:03:46.857917 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 03:03:46.865653 2702 catalog_manager.cc:1347] Generated new cluster ID: f0aa2791a2574d70ad76ddc762551828
I20260928 03:03:46.865758 2702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:46.882769 2702 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:46.884034 2702 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:46.897974 2702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c28e931feed147b69728d51cdeed8a76: Generated new TSK 0
I20260928 03:03:46.899060 2702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:46.910784 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2616
I20260928 03:03:46.911859 2678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:47.001890 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 2616
2026-09-28T03:03:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:47 dist-test-slave-k5rg krb5kdc[2715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:47 dist-test-slave-k5rg krb5kdc[2715](info): set up 2 sockets
Sep 28 03:03:47 dist-test-slave-k5rg krb5kdc[2715](info): commencing operation
krb5kdc: starting...
W20260928 03:03:48.017920 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:48 dist-test-slave-k5rg krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:03:48.254833 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41249
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:41249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:48.575278 2728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:48.575593 2728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:48.575726 2728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:48.583285 2728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:48.583441 2728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:48.583524 2728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:48.583629 2728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:48.583689 2728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:48.595594 2728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:41249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:41249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:48.597368 2728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:48.598721 2728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:48.609937 2734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:48.610240 2736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:48.609958 2733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:48.610765 2728 server_base.cc:1034] running on GCE node
I20260928 03:03:48.611475 2728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:48.612797 2728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:48.614018 2728 hybrid_clock.cc:648] HybridClock initialized: now 1790564628613958 us; error 52 us; skew 500 ppm
I20260928 03:03:48.614339 2728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:48 dist-test-slave-k5rg krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564628, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:48.617256 2728 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:03:48.618606 2728 webserver.cc:460] Webserver started at http://127.12.182.126:42773/ using document root <none> and password file <none>
I20260928 03:03:48.619238 2728 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:48.619351 2728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:48.619621 2728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:48.622768 2728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "b5052148986b4794b4c1f6de98d41de3"
format_stamp: "Formatted at 2026-09-28 03:03:48 on dist-test-slave-k5rg"
server_key: "f87c173928b5b0448418009471d6f1c6"
server_key_iv: "673f73b68596a86588b653a86607acc5"
server_key_version: "encryptionkey@0"
I20260928 03:03:48.623494 2728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "b5052148986b4794b4c1f6de98d41de3"
format_stamp: "Formatted at 2026-09-28 03:03:48 on dist-test-slave-k5rg"
server_key: "f87c173928b5b0448418009471d6f1c6"
server_key_iv: "673f73b68596a86588b653a86607acc5"
server_key_version: "encryptionkey@0"
I20260928 03:03:48.628504 2728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 03:03:48.631922 2743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:48.633265 2728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:03:48.633422 2728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "b5052148986b4794b4c1f6de98d41de3"
format_stamp: "Formatted at 2026-09-28 03:03:48 on dist-test-slave-k5rg"
server_key: "f87c173928b5b0448418009471d6f1c6"
server_key_iv: "673f73b68596a86588b653a86607acc5"
server_key_version: "encryptionkey@0"
I20260928 03:03:48.633623 2728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:48.650388 2728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:48.654556 2728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:48.654825 2728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:48.671648 2728 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:41249
I20260928 03:03:48.671674 2794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:41249 every 8 connection(s)
I20260928 03:03:48.673002 2728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:48.676530 2795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:48.677544 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 2728
I20260928 03:03:48.677718 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:48.678155 13017 external_mini_cluster.cc:1428] Setting key d2563d13029f9a6eae322abe5bfcdbec
I20260928 03:03:48.689049 2795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3: Bootstrap starting.
I20260928 03:03:48.690989 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:48.679888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60197 (local address 127.12.182.126:41249)
0928 03:03:48.680649 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:48.680669 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:48.680736 (+ 67us) server_negotiation.cc:407] Connection header received
0928 03:03:48.682765 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:48.682796 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:48.683306 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:48.683721 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:48.684219 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:48.685435 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:48.686696 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:48.687397 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:48.687555 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:48.688749 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:48.688756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:48.688771 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:48.689339 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:48.689422 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:48.689447 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:48.689549 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:48.689793 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 03:03:48.690051 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":289,"threads_started":1}
I20260928 03:03:48.692155 2795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:48.693177 2795 log.cc:824] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:48.694479 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:48.695791 2795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3: No bootstrap required, opened a new log
I20260928 03:03:48.703497 2795 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5052148986b4794b4c1f6de98d41de3" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41249 } }
I20260928 03:03:48.703905 2795 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:48.704018 2795 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5052148986b4794b4c1f6de98d41de3, State: Initialized, Role: FOLLOWER
I20260928 03:03:48.704312 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:48.698600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53317 (local address 127.12.182.126:41249)
0928 03:03:48.698781 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:48.698787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:48.699141 (+ 354us) server_negotiation.cc:407] Connection header received
0928 03:03:48.699325 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:48.699329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:48.699390 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:48.699469 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:48.699945 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:48.700682 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:48.702017 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:48.702469 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:48.702577 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:48.703609 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:48.703615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:48.703618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:48.703634 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:48.703670 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:48.703684 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:48.703769 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:48.703889 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 03:03:48.703975 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 03:03:48.704649 2795 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5052148986b4794b4c1f6de98d41de3" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41249 } }
I20260928 03:03:48.704924 2795 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:48.705077 2795 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:48.705255 2795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:48.707774 2795 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5052148986b4794b4c1f6de98d41de3" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41249 } }
I20260928 03:03:48.708223 2795 leader_election.cc:302] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5052148986b4794b4c1f6de98d41de3; no voters:
I20260928 03:03:48.708727 2795 leader_election.cc:288] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:48.708840 2805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:48.709436 2805 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [term 1 LEADER]: Becoming Leader. State: Replica: b5052148986b4794b4c1f6de98d41de3, State: Running, Role: LEADER
I20260928 03:03:48.709956 2805 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5052148986b4794b4c1f6de98d41de3" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41249 } }
I20260928 03:03:48.710610 2795 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:48.712822 2806 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5052148986b4794b4c1f6de98d41de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5052148986b4794b4c1f6de98d41de3" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41249 } } }
I20260928 03:03:48.713208 2806 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:48.713752 2807 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5052148986b4794b4c1f6de98d41de3. Latest consensus state: current_term: 1 leader_uuid: "b5052148986b4794b4c1f6de98d41de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5052148986b4794b4c1f6de98d41de3" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 41249 } } }
I20260928 03:03:48.714438 2807 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:48.715381 2812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:48.719936 2812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:48.729102 2812 catalog_manager.cc:1347] Generated new cluster ID: c38397f84884418ca7065f5b8ea35750
I20260928 03:03:48.729218 2812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:48.747545 2812 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:48.748795 2812 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:48.756170 2812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5052148986b4794b4c1f6de98d41de3: Generated new TSK 0
I20260928 03:03:48.756923 2812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:48.767241 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2728
W20260928 03:03:48.768594 2790 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60197
W20260928 03:03:48.769307 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2728: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:48.769428 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 2728
2026-09-28T03:03:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1766 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:48 dist-test-slave-k5rg krb5kdc[2826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:48 dist-test-slave-k5rg krb5kdc[2826](info): set up 2 sockets
Sep 28 03:03:48 dist-test-slave-k5rg krb5kdc[2826](info): commencing operation
krb5kdc: starting...
W20260928 03:03:49.777901 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:49 dist-test-slave-k5rg krb5kdc[2826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:03:50.013109 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:37177
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:42029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:37177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:50.325392 2839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:50.325696 2839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:50.325829 2839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:50.332722 2839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:50.332823 2839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:50.332895 2839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:50.332968 2839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:50.333025 2839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:50.342428 2839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:42029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:37177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:37177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:50.343658 2839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:50.344911 2839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:50.355892 2844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:50.355978 2845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:50.355942 2847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:50.357197 2839 server_base.cc:1034] running on GCE node
I20260928 03:03:50.357931 2839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:50.359230 2839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:50.360420 2839 hybrid_clock.cc:648] HybridClock initialized: now 1790564630360379 us; error 31 us; skew 500 ppm
I20260928 03:03:50.360744 2839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:50 dist-test-slave-k5rg krb5kdc[2826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564630, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:50.363593 2839 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:03:50.364971 2839 webserver.cc:460] Webserver started at http://127.12.182.126:38011/ using document root <none> and password file <none>
I20260928 03:03:50.365577 2839 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:50.365733 2839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:50.366015 2839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:50.368782 2839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "416e000ccb68473bb63a08c15e3e1521"
format_stamp: "Formatted at 2026-09-28 03:03:50 on dist-test-slave-k5rg"
server_key: "f214c98ae1358cc893c69bfee8ac05e5"
server_key_iv: "5e8bb3be1523e3b05d572b496e7ce147"
server_key_version: "encryptionkey@0"
I20260928 03:03:50.369488 2839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "416e000ccb68473bb63a08c15e3e1521"
format_stamp: "Formatted at 2026-09-28 03:03:50 on dist-test-slave-k5rg"
server_key: "f214c98ae1358cc893c69bfee8ac05e5"
server_key_iv: "5e8bb3be1523e3b05d572b496e7ce147"
server_key_version: "encryptionkey@0"
I20260928 03:03:50.374228 2839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 03:03:50.377336 2854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:50.378525 2839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 03:03:50.378690 2839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "416e000ccb68473bb63a08c15e3e1521"
format_stamp: "Formatted at 2026-09-28 03:03:50 on dist-test-slave-k5rg"
server_key: "f214c98ae1358cc893c69bfee8ac05e5"
server_key_iv: "5e8bb3be1523e3b05d572b496e7ce147"
server_key_version: "encryptionkey@0"
I20260928 03:03:50.378870 2839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:50.384279 2839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:50.387921 2839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:50.388175 2839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:50.404683 2839 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:37177
I20260928 03:03:50.404701 2905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:37177 every 8 connection(s)
I20260928 03:03:50.406008 2839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:50.409389 2906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:50.415282 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 2839
I20260928 03:03:50.415445 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:50.415942 13017 external_mini_cluster.cc:1428] Setting key d83ee3a0cb1fa6e2b9ecb1d4c2862fcf
I20260928 03:03:50.419822 2906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521: Bootstrap starting.
I20260928 03:03:50.423135 2906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:50.423151 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:50.417595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33393 (local address 127.12.182.126:37177)
0928 03:03:50.418407 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:50.418418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:50.418455 (+ 37us) server_negotiation.cc:407] Connection header received
0928 03:03:50.420075 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:50.420097 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:50.420418 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:50.420722 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:50.421754 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:50.421761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:50.421773 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:50.421791 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:50.421849 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:50.421868 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:50.422020 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:50.422125 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 03:03:50.422282 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":343,"threads_started":1}
I20260928 03:03:50.424178 2906 log.cc:824] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:50.426671 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:50.427027 2906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521: No bootstrap required, opened a new log
I20260928 03:03:50.432790 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:50.430495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40313 (local address 127.12.182.126:37177)
0928 03:03:50.430686 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:50.430692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:50.431118 (+ 426us) server_negotiation.cc:407] Connection header received
0928 03:03:50.431264 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:50.431268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:50.431315 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:50.431398 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:50.432236 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:50.432240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:50.432243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:03:50.432257 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:50.432296 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:03:50.432309 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:50.432397 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:50.432489 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 03:03:50.432531 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 03:03:50.435001 2906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416e000ccb68473bb63a08c15e3e1521" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37177 } }
I20260928 03:03:50.435354 2906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:03:50.435446 2906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416e000ccb68473bb63a08c15e3e1521, State: Initialized, Role: FOLLOWER
I20260928 03:03:50.435957 2906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416e000ccb68473bb63a08c15e3e1521" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37177 } }
I20260928 03:03:50.436215 2906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:03:50.436334 2906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:03:50.436492 2906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:50.438669 2906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416e000ccb68473bb63a08c15e3e1521" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37177 } }
I20260928 03:03:50.439142 2906 leader_election.cc:302] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 416e000ccb68473bb63a08c15e3e1521; no voters:
I20260928 03:03:50.439677 2906 leader_election.cc:288] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:03:50.439796 2916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:50.441368 2906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:50.445806 2916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [term 1 LEADER]: Becoming Leader. State: Replica: 416e000ccb68473bb63a08c15e3e1521, State: Running, Role: LEADER
I20260928 03:03:50.446270 2916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416e000ccb68473bb63a08c15e3e1521" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37177 } }
I20260928 03:03:50.448768 2917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "416e000ccb68473bb63a08c15e3e1521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416e000ccb68473bb63a08c15e3e1521" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37177 } } }
I20260928 03:03:50.449085 2917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:50.449393 2918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 416e000ccb68473bb63a08c15e3e1521. Latest consensus state: current_term: 1 leader_uuid: "416e000ccb68473bb63a08c15e3e1521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416e000ccb68473bb63a08c15e3e1521" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 37177 } } }
I20260928 03:03:50.453089 2918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521 [sys.catalog]: This master's current role is: LEADER
W20260928 03:03:50.457906 2930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:03:50.458024 2930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:03:50.458163 2931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:50.460251 2931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:50.468725 2931 catalog_manager.cc:1347] Generated new cluster ID: 00a4ec5787914643b3fd2771c908d959
I20260928 03:03:50.468842 2931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:50.492801 2931 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:50.493181 2931 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:50.501621 2931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 416e000ccb68473bb63a08c15e3e1521: Generated new TSK 0
I20260928 03:03:50.502692 2931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:50.514030 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2839
W20260928 03:03:50.515173 2901 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33393
W20260928 03:03:50.515805 13017 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2839: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:03:50.515930 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 2839
2026-09-28T03:03:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1746 ms)
[----------] 18 tests from AuthTokenIssuingTest (32351 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:50 dist-test-slave-k5rg krb5kdc[2937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:50 dist-test-slave-k5rg krb5kdc[2937](info): set up 2 sockets
Sep 28 03:03:50 dist-test-slave-k5rg krb5kdc[2937](info): commencing operation
krb5kdc: starting...
W20260928 03:03:51.534063 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:51 dist-test-slave-k5rg krb5kdc[2937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:03:51.776989 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33187
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33187,127.12.182.125:40459,127.12.182.124:33961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:52.092485 2950 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:03:52.092819 2950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:52.092922 2950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:52.093020 2950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:52.100308 2950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:52.100402 2950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:52.100469 2950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:52.100544 2950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:52.100615 2950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:52.110440 2950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33187,127.12.182.125:40459,127.12.182.124:33961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:52.111722 2950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:52.112989 2950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:52.123953 2955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:52.123991 2956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:52.123909 2958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:52.124933 2950 server_base.cc:1034] running on GCE node
I20260928 03:03:52.125699 2950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:52.127175 2950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:52.128400 2950 hybrid_clock.cc:648] HybridClock initialized: now 1790564632128333 us; error 42 us; skew 500 ppm
I20260928 03:03:52.128803 2950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:52 dist-test-slave-k5rg krb5kdc[2937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:52.136412 2950 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:03:52.138069 2950 webserver.cc:460] Webserver started at http://127.12.182.126:37549/ using document root <none> and password file <none>
I20260928 03:03:52.138792 2950 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:52.138890 2950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:52.139215 2950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:52.142330 2950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "74699677a7e04f0e884c3966b9231bbf"
format_stamp: "Formatted at 2026-09-28 03:03:52 on dist-test-slave-k5rg"
server_key: "9e0804918a3d9d96ba4ff7cf0a3be405"
server_key_iv: "0155bc913a21447f53361a96816e3fbf"
server_key_version: "encryptionkey@0"
I20260928 03:03:52.143157 2950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "74699677a7e04f0e884c3966b9231bbf"
format_stamp: "Formatted at 2026-09-28 03:03:52 on dist-test-slave-k5rg"
server_key: "9e0804918a3d9d96ba4ff7cf0a3be405"
server_key_iv: "0155bc913a21447f53361a96816e3fbf"
server_key_version: "encryptionkey@0"
I20260928 03:03:52.148497 2950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:03:52.152015 2965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:52.153353 2950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:03:52.153502 2950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "74699677a7e04f0e884c3966b9231bbf"
format_stamp: "Formatted at 2026-09-28 03:03:52 on dist-test-slave-k5rg"
server_key: "9e0804918a3d9d96ba4ff7cf0a3be405"
server_key_iv: "0155bc913a21447f53361a96816e3fbf"
server_key_version: "encryptionkey@0"
I20260928 03:03:52.153748 2950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:52.170429 2950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:52.174635 2950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:52.174911 2950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:52.192307 2950 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:33187
I20260928 03:03:52.192350 3016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:33187 every 8 connection(s)
I20260928 03:03:52.193646 2950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:52.197223 3017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:52.199458 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 2950
I20260928 03:03:52.199687 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:52.200414 13017 external_mini_cluster.cc:1428] Setting key b4222ebba017b7bc9065dde52011ce2f
I20260928 03:03:52.205777 3017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:03:52 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.125@KRBTEST.COM" created.
I20260928 03:03:52.227787 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.209437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.12.182.126:33187)
0928 03:03:52.211100 (+ 1663us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:52.211107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:52.211162 (+ 55us) server_negotiation.cc:407] Connection header received
0928 03:03:52.212647 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:52.212652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:52.212889 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:52.213074 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:52.214364 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.215529 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.216868 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.217395 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.217580 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:52.220908 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:52.220915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:52.220926 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:52.220958 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:52.223607 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:52.224261 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:52.224267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:52.224270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:52.224410 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:52.224770 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:52.224774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:52.224775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:52.225274 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:52.225464 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:52.226027 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 03:03:52.226326 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1443,"spinlock_wait_cycles":26496,"thread_start_us":1249,"threads_started":1}
I20260928 03:03:52.227767 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.209436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33187 (local address 127.0.0.1:46758)
0928 03:03:52.210320 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.210364 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:52.212296 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:52.213115 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:52.213122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:52.213835 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:52.214172 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:52.214189 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.215673 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.215677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:52.216738 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:52.216751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.217643 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.217650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:52.217816 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:52.218429 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:52.218450 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:52.220709 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:52.223723 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:52.223733 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:52.223745 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:52.224130 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:52.224514 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:52.224518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:52.224522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:52.224668 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:52.225409 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:52.225416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:52.225628 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 03:03:52.225851 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 03:03:52.226449 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":169,"thread_start_us":388,"threads_started":1}
I20260928 03:03:52.230749 3017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:03:52.231707 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.231353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:50930)
0928 03:03:52.231500 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.231591 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 03:03:52.231917 2966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
I20260928 03:03:52.233362 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.233024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:50932)
0928 03:03:52.233170 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.233234 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 03:03:52.233547 3017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:40459: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab.
Entry for principal kudu/127.12.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab.
I20260928 03:03:52.256769 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.125:40459
--webserver_interface=127.12.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab
--principal=kudu/127.12.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33187,127.12.182.125:40459,127.12.182.124:33961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:03:52.282898 3017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } attempt: 1
I20260928 03:03:52.284135 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.283638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:50942)
0928 03:03:52.283847 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.283979 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 03:03:52.285140 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.284761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:50958)
0928 03:03:52.284907 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.284992 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 03:03:52.285346 3017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:40459: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
I20260928 03:03:52.353633 3017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } attempt: 2
I20260928 03:03:52.354892 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.354337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:50964)
0928 03:03:52.354595 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.354719 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 03:03:52.355924 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.355511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:50972)
0928 03:03:52.355690 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.355774 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 03:03:52.356134 3017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:40459: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
I20260928 03:03:52.447378 3017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } attempt: 3
I20260928 03:03:52.448559 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.448072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:50978)
0928 03:03:52.448287 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.448398 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 03:03:52.449646 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.449198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:50984)
0928 03:03:52.449358 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.449444 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 03:03:52.449867 3017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:40459: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
W20260928 03:03:52.568792 3024 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:03:52.569120 3024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:52.569224 3024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:52.569344 3024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:52.577042 3024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:52.577142 3024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:52.577220 3024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:52.577291 3024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:52.577370 3024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:52.587394 3024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33187,127.12.182.125:40459,127.12.182.124:33961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab
--principal=kudu/127.12.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.125:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:52.588631 3024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:52.589877 3024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 03:03:52.593119 3017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } attempt: 4
I20260928 03:03:52.594372 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.593798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:50998)
0928 03:03:52.594107 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.594211 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260928 03:03:52.595583 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.595119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:51004)
0928 03:03:52.595334 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.595433 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260928 03:03:52.595907 3017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:40459: Network error: Client connection negotiation failed: client connection to 127.12.182.125:40459: connect: Connection refused (error 111)
W20260928 03:03:52.601063 3032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:52.601076 3029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:52.601082 3030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:52.602134 3024 server_base.cc:1034] running on GCE node
I20260928 03:03:52.602849 3024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:52.604386 3024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:52.605665 3024 hybrid_clock.cc:648] HybridClock initialized: now 1790564632605565 us; error 50 us; skew 500 ppm
I20260928 03:03:52.605994 3024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:52 dist-test-slave-k5rg krb5kdc[2937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:52.609022 3024 init.cc:377] Logged in from keytab as kudu/127.12.182.125@KRBTEST.COM (short username kudu)
I20260928 03:03:52.610419 3024 webserver.cc:460] Webserver started at http://127.12.182.125:39823/ using document root <none> and password file <none>
I20260928 03:03:52.611071 3024 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:52.611157 3024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:52.611442 3024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:52.614387 3024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/data/instance:
uuid: "22226f7db8504e58b137400c0949dd10"
format_stamp: "Formatted at 2026-09-28 03:03:52 on dist-test-slave-k5rg"
server_key: "2e9c61ac920729bcb65f19845eba1412"
server_key_iv: "f7c3cafc95131ee9fad4a32145e54b84"
server_key_version: "encryptionkey@0"
I20260928 03:03:52.615135 3024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/wal/instance:
uuid: "22226f7db8504e58b137400c0949dd10"
format_stamp: "Formatted at 2026-09-28 03:03:52 on dist-test-slave-k5rg"
server_key: "2e9c61ac920729bcb65f19845eba1412"
server_key_iv: "f7c3cafc95131ee9fad4a32145e54b84"
server_key_version: "encryptionkey@0"
I20260928 03:03:52.620194 3024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:03:52.623418 3039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:52.625509 3024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:03:52.625712 3024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/wal
uuid: "22226f7db8504e58b137400c0949dd10"
format_stamp: "Formatted at 2026-09-28 03:03:52 on dist-test-slave-k5rg"
server_key: "2e9c61ac920729bcb65f19845eba1412"
server_key_iv: "f7c3cafc95131ee9fad4a32145e54b84"
server_key_version: "encryptionkey@0"
I20260928 03:03:52.625907 3024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:52.663568 3024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:52.667577 3024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:52.667824 3024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:52.684564 3024 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.125:40459
I20260928 03:03:52.684567 3090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.125:40459 every 8 connection(s)
I20260928 03:03:52.685976 3024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
I20260928 03:03:52.688951 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 3024
I20260928 03:03:52.689126 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-1/wal/instance
I20260928 03:03:52.689527 3091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:52.689534 13017 external_mini_cluster.cc:1428] Setting key 04b64b86b82d03969c7533ae74903e38
I20260928 03:03:52.695801 3091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:03:52 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.124@KRBTEST.COM" created.
I20260928 03:03:52.715358 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.698535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.12.182.126:33187)
0928 03:03:52.698749 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:52.698755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:52.699834 (+ 1079us) server_negotiation.cc:407] Connection header received
0928 03:03:52.701661 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:52.701667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:52.701747 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:52.701869 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:52.703242 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.704193 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.705622 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.706231 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.706345 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:52.709754 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:52.709761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:52.709765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:52.709808 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:52.712424 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:52.713128 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:52.713133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:52.713136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:52.713262 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:52.713685 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:52.713689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:52.713691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:52.713974 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:52.714115 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:52.714821 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 03:03:52.715045 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:03:52.716282 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.698923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33187 (local address 127.0.0.1:42838)
0928 03:03:52.699667 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.699712 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:52.701402 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:52.702041 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:52.702048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:52.702549 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:52.703070 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:52.703086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.704320 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.704323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:52.705471 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:52.705482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.706441 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.706446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:52.706643 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:52.707310 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:52.707331 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:52.709509 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:52.712545 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:52.712552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:52.712565 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:52.713014 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:52.713373 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:52.713376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:52.713379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:52.713539 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:52.714107 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:52.714114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:52.714475 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 03:03:52.714718 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 03:03:52.715213 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":309,"threads_started":1}
I20260928 03:03:52.717986 3091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:03:52 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
I20260928 03:03:52.732115 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.718699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:40010)
0928 03:03:52.718860 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.718877 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:52.718983 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:52.719797 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:52.719800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:52.720128 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:52.720259 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:52.720267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.721789 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.721794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:52.722997 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:52.723011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.723810 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.723819 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:52.723994 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:52.724536 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:52.724556 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:52.726718 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:52.729780 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:52.729786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:52.729790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:52.730203 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:52.730534 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:52.730539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:52.730543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:52.730675 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:52.731219 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:52.731224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:52.731356 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 03:03:52.731467 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 03:03:52.731720 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 03:03:52.732174 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.718768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.12.182.125:40459)
0928 03:03:52.719322 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:52.719330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:52.719359 (+ 29us) server_negotiation.cc:407] Connection header received
0928 03:03:52.719407 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:52.719415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:52.719634 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:52.719771 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:52.720417 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.721650 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.723128 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.723672 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.723762 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:52.726902 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:52.726913 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:52.726926 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:52.726956 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:52.729567 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:52.730328 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:52.730334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:52.730336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:52.730440 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:52.730766 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:52.730771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:52.730774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:52.731098 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:52.731277 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:52.731613 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 03:03:52.731819 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":259,"threads_started":1}
I20260928 03:03:52.734422 3091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:03:52.735469 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.735036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33456)
0928 03:03:52.735197 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.735322 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 03:03:52.735687 3040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
I20260928 03:03:52.737064 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.736720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33470)
0928 03:03:52.736887 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.736966 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 03:03:52.737262 3091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:33961: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab.
Entry for principal kudu/127.12.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab.
I20260928 03:03:52.742710 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.124:33961
--webserver_interface=127.12.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab
--principal=kudu/127.12.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33187,127.12.182.125:40459,127.12.182.124:33961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:03:52.786579 3091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } attempt: 1
I20260928 03:03:52.787815 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.787293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33480)
0928 03:03:52.787520 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.787631 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 03:03:52.788966 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.788560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33490)
0928 03:03:52.788751 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.788839 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 03:03:52.789224 3091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:33961: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
I20260928 03:03:52.857457 3091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } attempt: 2
I20260928 03:03:52.858579 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.858115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33498)
0928 03:03:52.858320 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.858416 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 03:03:52.859637 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.859281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33506)
0928 03:03:52.859420 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.859493 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 03:03:52.859849 3091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:33961: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
I20260928 03:03:52.895174 3017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } attempt: 5
Sep 28 03:03:52 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
I20260928 03:03:52.909562 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.895866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:40012)
0928 03:03:52.896124 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.896149 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:52.896320 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:52.896674 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:52.896679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:52.897241 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:52.897508 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:52.897524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.898608 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.898614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:52.900032 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:52.900050 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.900875 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.900880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:52.901115 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:52.901853 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:52.901877 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:52.904295 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:52.907190 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:52.907196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:52.907200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:52.907565 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:52.907915 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:52.907921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:52.907924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:52.908060 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:52.908596 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:52.908601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:52.908768 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 03:03:52.908887 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 03:03:52.909187 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 03:03:52.909562 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.896117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.12.182.125:40459)
0928 03:03:52.896295 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:52.896299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:52.896319 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:03:52.896441 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:52.896445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:52.896520 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:52.896601 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:52.897749 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.898461 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.900223 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:52.900720 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:52.900825 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:52.904501 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:52.904506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:52.904509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:52.904545 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:52.907043 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:52.907683 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:52.907687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:52.907689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:52.907809 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:52.908174 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:52.908180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:52.908182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:52.908479 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:52.908623 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:52.909008 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 03:03:52.909232 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 03:03:52.910739 3017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:03:52.911883 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.911320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33516)
0928 03:03:52.911635 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.911733 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 03:03:52.912854 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.912523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33528)
0928 03:03:52.912681 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.912754 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 03:03:52.913050 3017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:33961: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
I20260928 03:03:52.951067 3091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } attempt: 3
I20260928 03:03:52.952225 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.951754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33530)
0928 03:03:52.951972 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.952057 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 03:03:52.953271 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.952868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33538)
0928 03:03:52.953072 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.953162 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260928 03:03:52.953469 3091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:33961: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
I20260928 03:03:52.964263 3017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } attempt: 1
I20260928 03:03:52.965281 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.964846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33540)
0928 03:03:52.965030 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.965111 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 03:03:52.966178 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:52.965871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33554)
0928 03:03:52.966008 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:52.966072 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 03:03:52.966354 3017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:33961: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
I20260928 03:03:53.034576 3017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } attempt: 2
I20260928 03:03:53.035809 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.035311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33564)
0928 03:03:53.035547 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.035646 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 03:03:53.036839 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.036480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33580)
0928 03:03:53.036637 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.036723 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 03:03:53.037039 3017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:33961: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
W20260928 03:03:53.064011 3098 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:03:53.064339 3098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:53.064431 3098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:53.064538 3098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:53.071779 3098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:53.071878 3098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:53.071957 3098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:53.072021 3098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:53.072101 3098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:53.081858 3098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33187,127.12.182.125:40459,127.12.182.124:33961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab
--principal=kudu/127.12.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.124:33961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:53.083104 3098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:53.084380 3098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:53.095489 3103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:53.095517 3104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:53.095610 3106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:53.096110 3098 server_base.cc:1034] running on GCE node
I20260928 03:03:53.096653 3091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } attempt: 4
I20260928 03:03:53.096905 3098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:53.097752 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.097240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33584)
0928 03:03:53.097438 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.097521 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 03:03:53.098425 3098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:53.098825 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.098431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33586)
0928 03:03:53.098589 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.098702 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 03:03:53.099087 3091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:33961: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
I20260928 03:03:53.099661 3098 hybrid_clock.cc:648] HybridClock initialized: now 1790564633099585 us; error 53 us; skew 500 ppm
I20260928 03:03:53.099946 3098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:53 dist-test-slave-k5rg krb5kdc[2937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:53.102978 3098 init.cc:377] Logged in from keytab as kudu/127.12.182.124@KRBTEST.COM (short username kudu)
I20260928 03:03:53.104313 3098 webserver.cc:460] Webserver started at http://127.12.182.124:46771/ using document root <none> and password file <none>
I20260928 03:03:53.104943 3098 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:53.105026 3098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:53.105309 3098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:53.108156 3098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/data/instance:
uuid: "2f251a6d6a4d4971b8617c9bbede8fa9"
format_stamp: "Formatted at 2026-09-28 03:03:53 on dist-test-slave-k5rg"
server_key: "66db6122a0550e82a08dca97201af892"
server_key_iv: "64b16e8a8c5255b19dd9f8db2996bb5d"
server_key_version: "encryptionkey@0"
I20260928 03:03:53.108891 3098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/wal/instance:
uuid: "2f251a6d6a4d4971b8617c9bbede8fa9"
format_stamp: "Formatted at 2026-09-28 03:03:53 on dist-test-slave-k5rg"
server_key: "66db6122a0550e82a08dca97201af892"
server_key_iv: "64b16e8a8c5255b19dd9f8db2996bb5d"
server_key_version: "encryptionkey@0"
I20260928 03:03:53.113803 3098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 03:03:53.117108 3113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:53.118417 3098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:03:53.118566 3098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/wal
uuid: "2f251a6d6a4d4971b8617c9bbede8fa9"
format_stamp: "Formatted at 2026-09-28 03:03:53 on dist-test-slave-k5rg"
server_key: "66db6122a0550e82a08dca97201af892"
server_key_iv: "64b16e8a8c5255b19dd9f8db2996bb5d"
server_key_version: "encryptionkey@0"
I20260928 03:03:53.118777 3098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:53.135881 3098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:53.140133 3098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:53.140348 3098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:53.143244 3017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } attempt: 3
I20260928 03:03:53.144400 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.143928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33594)
0928 03:03:53.144142 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.144228 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 03:03:53.145336 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.145016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33604)
0928 03:03:53.145169 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.145235 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 03:03:53.145697 3017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:33961: Network error: Client connection negotiation failed: client connection to 127.12.182.124:33961: connect: Connection refused (error 111)
I20260928 03:03:53.157505 3098 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.124:33961
I20260928 03:03:53.157523 3164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.124:33961 every 8 connection(s)
I20260928 03:03:53.158941 3098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
I20260928 03:03:53.162719 3165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:53.166203 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 3098
I20260928 03:03:53.166368 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564295078766-13017-0/minicluster-data/master-2/wal/instance
I20260928 03:03:53.166805 13017 external_mini_cluster.cc:1428] Setting key 4cf14b088a7f24a88aa7e0bd0a30d2b8
I20260928 03:03:53.169888 3165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:03:53 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
Sep 28 03:03:53 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:53.187991 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.173336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.12.182.126:33187)
0928 03:03:53.173557 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.173563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.174286 (+ 723us) server_negotiation.cc:407] Connection header received
0928 03:03:53.174470 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.174476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.174553 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.174651 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.175839 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.176507 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.178047 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.178460 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.178564 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.181782 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.181788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.181793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.181832 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.184862 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.185664 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.185669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.185671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.185827 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.186192 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.186197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.186200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.186518 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.186678 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.187224 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.187638 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 03:03:53.188658 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.168603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33608 (local address 127.12.182.124:33961)
0928 03:03:53.169344 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.169356 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.169396 (+ 40us) server_negotiation.cc:407] Connection header received
0928 03:03:53.171030 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.171055 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.171482 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.171844 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.173308 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.174986 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.176238 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.176900 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.177069 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.181074 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.181084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.181095 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.181130 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.184407 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.185160 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.185166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.185168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.185320 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.185734 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.185738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.185741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.186425 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.186712 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.187036 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.187556 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":544,"thread_start_us":347,"threads_started":1}
I20260928 03:03:53.188691 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.173391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33187 (local address 127.0.0.1:42840)
0928 03:03:53.174142 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.174187 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:53.174350 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:53.174737 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:53.174742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:53.175349 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:53.175678 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:53.175688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.176745 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.176750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:53.177912 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:53.177923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.178674 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.178681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:53.178815 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.179351 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:53.179372 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:53.181591 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:53.185004 (+ 3413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:53.185017 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:53.185030 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:53.185479 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:53.185930 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:53.185933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:53.185935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:53.186085 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:53.186689 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:53.186697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:53.186905 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 03:03:53.187087 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 03:03:53.187652 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":174,"thread_start_us":364,"threads_started":1}
I20260928 03:03:53.190200 3165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:03:53 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
Sep 28 03:03:53 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:53.204490 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.190940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:40459 (local address 127.0.0.1:40014)
0928 03:03:53.191214 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.191232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:53.191345 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:53.191968 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:53.191973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:53.192383 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:53.192582 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:53.192595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.194012 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.194018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:53.195123 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:53.195136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.195829 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.195835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:53.196031 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.196887 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:53.196909 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:53.198969 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:53.202216 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:53.202222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:53.202226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:53.202593 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:53.202937 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:53.202942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:53.202946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:53.203083 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:53.203585 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:53.203591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:53.203746 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 03:03:53.203864 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 03:03:53.204148 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260928 03:03:53.204517 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.190940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.12.182.125:40459)
0928 03:03:53.191124 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.191130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.191293 (+ 163us) server_negotiation.cc:407] Connection header received
0928 03:03:53.191451 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.191456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.191535 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.191681 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.192751 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.193855 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.195262 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.195697 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.195806 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.199153 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.199158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.199161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.199195 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.202094 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.202699 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.202705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.202709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.202833 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.203179 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.203184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.203188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.203448 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.203599 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.203979 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.204185 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:03:53.205466 3165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:03:53.206681 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.191676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.12.182.126:33187)
0928 03:03:53.191848 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.191853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.191873 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:03:53.192055 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.192060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.192126 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.192216 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.193471 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.194503 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.196153 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.196703 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.196821 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.201681 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.201687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.201690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.201723 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.204127 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.204772 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.204777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.204780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.204928 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.205311 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.205315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.205317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.205565 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.205913 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.206176 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.206383 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 03:03:53 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564633, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
Sep 28 03:03:53 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
I20260928 03:03:53.218346 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.206099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33620)
0928 03:03:53.206239 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.206255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:53.206344 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:53.206699 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:53.206708 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:53.207268 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:53.207398 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:53.207406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.208573 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.208578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:53.209493 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:53.209504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.210281 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.210287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:53.210500 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.211073 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:53.211094 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:53.213172 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:53.215936 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:53.215942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:53.215946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:53.216420 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:53.216768 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:53.216771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:53.216774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:53.216899 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:53.217454 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:53.217459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:53.217642 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 03:03:53.217794 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 03:03:53.218071 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 03:03:53.218612 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.206299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.12.182.124:33961)
0928 03:03:53.206435 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.206440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.206454 (+ 14us) server_negotiation.cc:407] Connection header received
0928 03:03:53.206489 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.206493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.206555 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.206643 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.207537 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.208465 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.209675 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.210141 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.210241 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.213339 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.213343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.213346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.213373 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.215798 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.216525 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.216529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.216531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.216644 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.216998 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.217003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.217007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.217340 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.217463 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.217904 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.218249 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":49}
I20260928 03:03:53.222530 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.208153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.12.182.125:40459)
0928 03:03:53.208302 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.208307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.208340 (+ 33us) server_negotiation.cc:407] Connection header received
0928 03:03:53.208491 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.208496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.208572 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.208703 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.209820 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.210689 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.211977 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.212387 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.212493 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.216807 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.216813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.216818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.216849 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.219841 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.220647 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.220650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.220652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.220787 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.221177 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.221180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.221182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.221419 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.221666 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.221955 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.222253 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:03:53.224455 3165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9: Bootstrap starting.
I20260928 03:03:53.227290 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:53.228324 3165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:53.229621 3165 log.cc:824] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:53.233052 3165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9: No bootstrap required, opened a new log
I20260928 03:03:53.230921 2992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "74699677a7e04f0e884c3966b9231bbf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42850
I20260928 03:03:53.243728 3165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } }
I20260928 03:03:53.244180 3165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f251a6d6a4d4971b8617c9bbede8fa9, State: Initialized, Role: FOLLOWER
I20260928 03:03:53.244692 3165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } }
I20260928 03:03:53.245594 3171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } } }
I20260928 03:03:53.245951 3171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:53.246938 3165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 03:03:53.256047 3182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:03:53.256178 3182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:03:53.254262 3098 master_runner.cc:186] Error getting master registration for 127.12.182.126:33187: OK, instance_id { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" instance_seqno: 1790564632187213 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:03:53.265007 3098 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 03:03:53.322907 3017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } attempt: 4
Sep 28 03:03:53 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
I20260928 03:03:53.338855 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.323707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33622)
0928 03:03:53.323909 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.323930 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:53.324061 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:53.324471 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:53.324476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:53.325025 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:53.325232 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:53.325245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.326710 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.326716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:53.328150 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:53.328165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.329174 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.329180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:53.329382 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.330087 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:53.330104 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:53.332316 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:53.336134 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:53.336139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:53.336144 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:53.336645 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:53.337008 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:53.337012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:53.337016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:53.337179 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:53.337865 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:53.337869 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:53.338023 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 03:03:53.338169 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 03:03:53.338470 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 03:03:53.338879 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.323781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.12.182.124:33961)
0928 03:03:53.323959 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.323965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.323989 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:03:53.324236 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.324243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.324332 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.324449 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.325390 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.326577 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.328296 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.329022 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.329157 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.332496 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.332503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.332508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.332548 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.335992 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.336755 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.336761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.336764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.336911 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.337275 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.337280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.337284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.337749 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.337917 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.338278 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.338512 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:03:53.344916 3017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf: Bootstrap starting.
I20260928 03:03:53.349086 3017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:53.350453 3017 log.cc:824] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:53.353863 3017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf: No bootstrap required, opened a new log
I20260928 03:03:53.361465 3017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } }
I20260928 03:03:53.361907 3017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74699677a7e04f0e884c3966b9231bbf, State: Initialized, Role: FOLLOWER
I20260928 03:03:53.362423 3017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } }
I20260928 03:03:53.363449 3185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } } }
I20260928 03:03:53.363853 3185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:53.364738 3017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:53.364845 2992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 03:03:53.365027 2992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:53.367466 2992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } }
I20260928 03:03:53.368781 2992 leader_election.cc:288] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [CANDIDATE]: Term 1 election: Requested vote from peers 22226f7db8504e58b137400c0949dd10 (127.12.182.125:40459), 2f251a6d6a4d4971b8617c9bbede8fa9 (127.12.182.124:33961)
I20260928 03:03:53.370014 3140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74699677a7e04f0e884c3966b9231bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9"
I20260928 03:03:53.370492 3140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:53.373812 3140 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74699677a7e04f0e884c3966b9231bbf in term 1.
I20260928 03:03:53.375773 2966 leader_election.cc:302] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f251a6d6a4d4971b8617c9bbede8fa9, 74699677a7e04f0e884c3966b9231bbf; no voters:
I20260928 03:03:53.376228 3185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:53.389340 3185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [term 1 LEADER]: Becoming Leader. State: Replica: 74699677a7e04f0e884c3966b9231bbf, State: Running, Role: LEADER
I20260928 03:03:53.390734 3185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } }
W20260928 03:03:53.392338 3119 tablet.cc:2369] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:03:53.398275 3091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } attempt: 5
Sep 28 03:03:53 dist-test-slave-k5rg krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
I20260928 03:03:53.415841 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.398946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:33961 (local address 127.0.0.1:33634)
0928 03:03:53.399357 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:53.399377 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:53.399492 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:53.399983 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:53.399988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:53.400491 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:53.400712 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:53.400726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.402290 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.402296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:53.403697 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:53.403713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.404846 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.404853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:53.405090 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.405756 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:53.405780 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:53.408648 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:53.412932 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:53.412938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:53.412943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:53.413467 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:53.413949 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:53.413954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:53.413958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:53.414175 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:53.414810 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:53.414816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:53.414976 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 03:03:53.415116 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 03:03:53.415429 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271}
I20260928 03:03:53.416980 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.399346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.12.182.124:33961)
0928 03:03:53.399503 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.399508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.399529 (+ 21us) server_negotiation.cc:407] Connection header received
0928 03:03:53.399604 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.399608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.399702 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.399794 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.400903 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.402113 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.403870 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.404502 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.404636 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.408873 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.408880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.408885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.408931 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.412761 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.413646 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.413651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.413654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.413822 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.414322 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.414326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.414330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.414674 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.416244 (+ 1570us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.416372 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.416617 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 03:03:53.423035 3091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10: Bootstrap starting.
I20260928 03:03:53.427260 3091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10: Neither blocks nor log segments found. Creating new log.
W20260928 03:03:53.427721 2971 tablet.cc:2369] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:03:53.430943 3091 log.cc:824] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:53.433063 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.400133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.12.182.126:33187)
0928 03:03:53.400306 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.400311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.408378 (+ 8067us) server_negotiation.cc:407] Connection header received
0928 03:03:53.408869 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.408877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.408969 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.409041 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.419244 (+ 10203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.420335 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.422161 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.422849 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.422949 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.426204 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.426210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.426214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.426256 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.429795 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.430599 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.430604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.430608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.430811 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.431278 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.431282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.431285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.431633 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.431886 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.432498 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.432727 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:03:53.434402 3091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10: No bootstrap required, opened a new log
I20260928 03:03:53.450502 3193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74699677a7e04f0e884c3966b9231bbf. Latest consensus state: current_term: 1 leader_uuid: "74699677a7e04f0e884c3966b9231bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } } }
I20260928 03:03:53.450934 3193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:53.451297 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.433111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.12.182.124:33961)
0928 03:03:53.433545 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.433551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.435201 (+ 1650us) server_negotiation.cc:407] Connection header received
0928 03:03:53.435446 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.435452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.435539 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.435608 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.437409 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.438430 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.440230 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.440864 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.440961 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.444246 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.444253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.444259 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.444300 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.447947 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.448798 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.448803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.448807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.448958 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.449430 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.449439 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.449442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.449827 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.450069 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.450654 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.450898 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260928 03:03:53.453838 3091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } }
I20260928 03:03:53.454245 3091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22226f7db8504e58b137400c0949dd10, State: Initialized, Role: FOLLOWER
I20260928 03:03:53.454736 3207 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 03:03:53.454761 3091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } }
I20260928 03:03:53.455857 3206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:53.467453 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:53.402274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.12.182.125:40459)
0928 03:03:53.402446 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.402452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.435439 (+ 32987us) server_negotiation.cc:407] Connection header received
0928 03:03:53.436285 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.436290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.436368 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.436442 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.443970 (+ 7528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.445005 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.456765 (+ 11760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:53.457458 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:53.457556 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:53.460735 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:53.460740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:53.460743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:53.460773 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:53.464222 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.465029 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.465035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.465038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.465210 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:53.465639 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:53.465644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:53.465647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:53.465941 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:53.466185 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:53.466625 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 03:03:53.467072 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:03:53.458952 2950 master_runner.cc:186] Error getting master registration for 127.12.182.125:40459: OK, instance_id { permanent_uuid: "22226f7db8504e58b137400c0949dd10" instance_seqno: 1790564632679728 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:03:53.475903 2950 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 03:03:53.485227 3206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:53.489683 3208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } } }
I20260928 03:03:53.490268 3208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:53.491204 3091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:53.492077 3066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74699677a7e04f0e884c3966b9231bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "22226f7db8504e58b137400c0949dd10"
I20260928 03:03:53.492585 3066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:53.495841 3066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74699677a7e04f0e884c3966b9231bbf in term 1.
I20260928 03:03:53.505770 3140 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [term 1 FOLLOWER]: Refusing update from remote peer 74699677a7e04f0e884c3966b9231bbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:53.506779 3193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:53.516921 3066 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [term 1 FOLLOWER]: Refusing update from remote peer 74699677a7e04f0e884c3966b9231bbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:53.517738 3193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:53.542866 3208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74699677a7e04f0e884c3966b9231bbf. Latest consensus state: current_term: 1 leader_uuid: "74699677a7e04f0e884c3966b9231bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } } }
I20260928 03:03:53.543212 3208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:53.548029 3210 mvcc.cc:205] Tried to move back new op lower bound from 7334152738827550720 to 7334152738380730368. Current Snapshot: MvccSnapshot[applied={T|T < 7334152738827550720}]
I20260928 03:03:53.557264 3208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74699677a7e04f0e884c3966b9231bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } } }
I20260928 03:03:53.557541 3208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:53.562487 3193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74699677a7e04f0e884c3966b9231bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } } }
I20260928 03:03:53.562753 3193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:53.563424 3206 catalog_manager.cc:1347] Generated new cluster ID: b97d2c9d33e341ebb4bc4c36dfbffa29
I20260928 03:03:53.563519 3206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:53.565828 3171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74699677a7e04f0e884c3966b9231bbf. Latest consensus state: current_term: 1 leader_uuid: "74699677a7e04f0e884c3966b9231bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } } }
I20260928 03:03:53.566124 3171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:53.586127 3171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74699677a7e04f0e884c3966b9231bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } } }
I20260928 03:03:53.586383 3171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:53.587568 3185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74699677a7e04f0e884c3966b9231bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74699677a7e04f0e884c3966b9231bbf" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33187 } } peers { permanent_uuid: "22226f7db8504e58b137400c0949dd10" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 40459 } } peers { permanent_uuid: "2f251a6d6a4d4971b8617c9bbede8fa9" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 33961 } } }
I20260928 03:03:53.588022 3185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf [sys.catalog]: This master's current role is: LEADER
W20260928 03:03:53.594123 3221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:03:53.599779 3221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:03:53.621115 3206 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:53.623194 3206 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:03:53.654065 3206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74699677a7e04f0e884c3966b9231bbf: Generated new TSK 0
I20260928 03:03:53.655191 3206 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 03:03:53.803855 3020 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.789743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.12.182.126:33187)
0928 03:03:53.790007 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.790014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.801955 (+ 11941us) server_negotiation.cc:407] Connection header received
0928 03:03:53.802272 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.802279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.802366 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.802438 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.803612 (+ 1174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42868: BlockingRecv error: recv got EOF from 127.0.0.1:42868 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260928 03:03:53.806196 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.791802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.12.182.125:40459)
0928 03:03:53.791993 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.791999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.804597 (+ 12598us) server_negotiation.cc:407] Connection header received
0928 03:03:53.804820 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.804825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.804920 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.804989 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.805948 (+ 959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40026: BlockingRecv error: recv got EOF from 127.0.0.1:40026 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 03:03:53.808305 3168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.802032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.12.182.124:33961)
0928 03:03:53.802217 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.802223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.806823 (+ 4600us) server_negotiation.cc:407] Connection header received
0928 03:03:53.807051 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.807056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.807137 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.807204 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.808092 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33658: BlockingRecv error: recv got EOF from 127.0.0.1:33658 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 03:03:53.817418 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.815406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.12.182.125:40459)
0928 03:03:53.815571 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.815576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.815988 (+ 412us) server_negotiation.cc:407] Connection header received
0928 03:03:53.816208 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.816214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.816326 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.816428 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.817186 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40034: BlockingRecv error: recv got EOF from 127.0.0.1:40034 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 03:03:53.827445 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.821897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.12.182.125:40459)
0928 03:03:53.822061 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.822067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.826110 (+ 4043us) server_negotiation.cc:407] Connection header received
0928 03:03:53.826312 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.826322 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.826407 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.826505 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.827242 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40048: BlockingRecv error: recv got EOF from 127.0.0.1:40048 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 03:03:53.837114 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.835221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.12.182.125:40459)
0928 03:03:53.835363 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.835370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.835834 (+ 464us) server_negotiation.cc:407] Connection header received
0928 03:03:53.835992 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.835998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.836084 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.836187 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.836876 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40054: BlockingRecv error: recv got EOF from 127.0.0.1:40054 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 03:03:53.846208 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.844150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.12.182.125:40459)
0928 03:03:53.844319 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.844324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.844768 (+ 444us) server_negotiation.cc:407] Connection header received
0928 03:03:53.844951 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.844956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.845050 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.845154 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.845971 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40070: BlockingRecv error: recv got EOF from 127.0.0.1:40070 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 03:03:53.859604 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.857647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.12.182.125:40459)
0928 03:03:53.857848 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.857854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.858218 (+ 364us) server_negotiation.cc:407] Connection header received
0928 03:03:53.858378 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.858384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.858472 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.858574 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.859372 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40080: BlockingRecv error: recv got EOF from 127.0.0.1:40080 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 03:03:53.881494 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.879130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.12.182.125:40459)
0928 03:03:53.879318 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.879324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.879871 (+ 547us) server_negotiation.cc:407] Connection header received
0928 03:03:53.880090 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.880096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.880198 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.880309 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.881242 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40086: BlockingRecv error: recv got EOF from 127.0.0.1:40086 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 03:03:53.920737 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.918364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.12.182.125:40459)
0928 03:03:53.918559 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.918566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.919152 (+ 586us) server_negotiation.cc:407] Connection header received
0928 03:03:53.919386 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.919392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.919496 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.919608 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.920483 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40092: BlockingRecv error: recv got EOF from 127.0.0.1:40092 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 03:03:53.989786 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:53.987956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.12.182.125:40459)
0928 03:03:53.988158 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:53.988164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:53.988398 (+ 234us) server_negotiation.cc:407] Connection header received
0928 03:03:53.988587 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:53.988593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:53.988685 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:53.988814 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:53.989513 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40100: BlockingRecv error: recv got EOF from 127.0.0.1:40100 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 03:03:54.124437 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:54.122657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.12.182.125:40459)
0928 03:03:54.122837 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:54.122843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:54.123175 (+ 332us) server_negotiation.cc:407] Connection header received
0928 03:03:54.123367 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:54.123371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:54.123442 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:54.123537 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:54.124248 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40112: BlockingRecv error: recv got EOF from 127.0.0.1:40112 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 03:03:54.258667 3182 catalog_manager.cc:1259] Loaded cluster ID: b97d2c9d33e341ebb4bc4c36dfbffa29
I20260928 03:03:54.258766 3182 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9: loading cluster ID for follower catalog manager: success
I20260928 03:03:54.261310 3182 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9: acquiring CA information for follower catalog manager: success
I20260928 03:03:54.263064 3182 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f251a6d6a4d4971b8617c9bbede8fa9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 03:03:54.387151 3095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:54.384766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40116 (local address 127.12.182.125:40459)
0928 03:03:54.385002 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:54.385007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:54.385249 (+ 242us) server_negotiation.cc:407] Connection header received
0928 03:03:54.385406 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:54.385410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:54.385493 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:54.385723 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:54.386885 (+ 1162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40116: BlockingRecv error: recv got EOF from 127.0.0.1:40116 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 03:03:54.601982 3221 catalog_manager.cc:1259] Loaded cluster ID: b97d2c9d33e341ebb4bc4c36dfbffa29
I20260928 03:03:54.602088 3221 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10: loading cluster ID for follower catalog manager: success
I20260928 03:03:54.604785 3221 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10: acquiring CA information for follower catalog manager: success
I20260928 03:03:54.606714 3221 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22226f7db8504e58b137400c0949dd10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:03:54.912551 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:54.906976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.12.182.125:40459)
0928 03:03:54.907606 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:54.907611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:54.907628 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:03:54.907689 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:54.907693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:54.907836 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:54.907940 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:54.908412 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:54.909187 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:54.910485 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:54.910872 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:54.910963 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:54.911433 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:54.911963 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:54.912076 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:54.912202 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 03:03:54.912268 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":426,"threads_started":1}
I20260928 03:03:54.923821 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:54.918651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.12.182.124:33961)
0928 03:03:54.919159 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:54.919164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:54.919181 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:03:54.919303 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:54.919307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:54.919456 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:54.919592 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:54.919889 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:54.920579 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:54.921810 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:54.922209 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:54.922306 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:54.922727 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:54.923234 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:54.923343 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:54.923416 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 03:03:54.923501 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":291,"threads_started":1}
I20260928 03:03:54.939975 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:54.930070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.12.182.126:33187)
0928 03:03:54.934069 (+ 3999us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:54.934073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:54.934090 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:03:54.934176 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:54.934180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:54.934318 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:54.934466 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:54.935017 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:54.935997 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:54.937531 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:54.938108 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:54.938234 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:54.938857 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:54.939295 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:54.939403 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:54.939593 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 03:03:54.939677 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3904,"thread_start_us":3689,"threads_started":1}
I20260928 03:03:54.941411 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:54.935299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.12.182.124:33961)
0928 03:03:54.935457 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:54.935463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:54.935689 (+ 226us) server_negotiation.cc:407] Connection header received
0928 03:03:54.935830 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:54.935835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:54.935897 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:54.935968 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:54.936633 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:54.937747 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:54.939524 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:54.940066 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:54.940176 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:54.940753 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:54.940845 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:54.940909 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:54.941074 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 03:03:54.941156 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:03:54.942972 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:54.929869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.12.182.125:40459)
0928 03:03:54.930056 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:54.930063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:54.935499 (+ 5436us) server_negotiation.cc:407] Connection header received
0928 03:03:54.935681 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:54.935687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:54.935768 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:54.935850 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:54.936681 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:54.937762 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:54.939835 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:54.940365 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:54.940485 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:54.941437 (+ 952us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:54.941937 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:54.942030 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:54.942158 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 03:03:54.942708 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:03:54.945214 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 2950
I20260928 03:03:54.946254 3012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:55.056986 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 2950
I20260928 03:03:55.074088 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 3024
I20260928 03:03:55.075114 3086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:55.179917 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 3024
I20260928 03:03:55.196184 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 3098
I20260928 03:03:55.197338 3160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:55.304466 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 3098
2026-09-28T03:03:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4792 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:03:55 dist-test-slave-k5rg krb5kdc[3315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:03:55 dist-test-slave-k5rg krb5kdc[3315](info): set up 2 sockets
Sep 28 03:03:55 dist-test-slave-k5rg krb5kdc[3315](info): commencing operation
krb5kdc: starting...
W20260928 03:03:56.326028 13017 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/test-user.keytab.
Sep 28 03:03:56 dist-test-slave-k5rg krb5kdc[3315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:03:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:03:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
Entry for principal kudu/127.12.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab.
I20260928 03:03:56.566740 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33777
--webserver_interface=127.12.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33777,127.12.182.125:44709,127.12.182.124:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:03:56.878521 3328 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:03:56.878873 3328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:56.878968 3328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:56.879076 3328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:56.886576 3328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:56.886684 3328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:56.886763 3328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:56.886828 3328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:56.886911 3328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:56.896629 3328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33777,127.12.182.125:44709,127.12.182.124:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.126.keytab
--principal=kudu/127.12.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.126:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:56.897915 3328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:56.899149 3328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:03:56.910240 3333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:56.910211 3334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:56.910280 3336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:56.911276 3328 server_base.cc:1034] running on GCE node
I20260928 03:03:56.912115 3328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:56.913527 3328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:56.914798 3328 hybrid_clock.cc:648] HybridClock initialized: now 1790564636914719 us; error 46 us; skew 500 ppm
I20260928 03:03:56.915135 3328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:56 dist-test-slave-k5rg krb5kdc[3315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:56.918286 3328 init.cc:377] Logged in from keytab as kudu/127.12.182.126@KRBTEST.COM (short username kudu)
I20260928 03:03:56.919863 3328 webserver.cc:460] Webserver started at http://127.12.182.126:43685/ using document root <none> and password file <none>
I20260928 03:03:56.920567 3328 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:56.920680 3328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:56.920975 3328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:56.923986 3328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/data/instance:
uuid: "03aee46f358f402da7f4acbfbf1c95c8"
format_stamp: "Formatted at 2026-09-28 03:03:56 on dist-test-slave-k5rg"
server_key: "3f084341e82fc15a00d5f18443d444ae"
server_key_iv: "74e7eea4bdaf8a7d0ced9692544533df"
server_key_version: "encryptionkey@0"
I20260928 03:03:56.924767 3328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/wal/instance:
uuid: "03aee46f358f402da7f4acbfbf1c95c8"
format_stamp: "Formatted at 2026-09-28 03:03:56 on dist-test-slave-k5rg"
server_key: "3f084341e82fc15a00d5f18443d444ae"
server_key_iv: "74e7eea4bdaf8a7d0ced9692544533df"
server_key_version: "encryptionkey@0"
I20260928 03:03:56.929867 3328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 03:03:56.933251 3343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:56.934587 3328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:03:56.934749 3328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/wal
uuid: "03aee46f358f402da7f4acbfbf1c95c8"
format_stamp: "Formatted at 2026-09-28 03:03:56 on dist-test-slave-k5rg"
server_key: "3f084341e82fc15a00d5f18443d444ae"
server_key_iv: "74e7eea4bdaf8a7d0ced9692544533df"
server_key_version: "encryptionkey@0"
I20260928 03:03:56.934953 3328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:56.964537 3328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:56.968497 3328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:56.968760 3328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:56.985044 3328 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.126:33777
I20260928 03:03:56.985050 3394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.126:33777 every 8 connection(s)
I20260928 03:03:56.986338 3328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/data/info.pb
I20260928 03:03:56.988745 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 3328
I20260928 03:03:56.988909 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-0/wal/instance
I20260928 03:03:56.989322 13017 external_mini_cluster.cc:1428] Setting key 1522696bc205eb702affdbae69fe6e84
I20260928 03:03:56.989921 3395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:56.996062 3395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.125@KRBTEST.COM" created.
I20260928 03:03:57.018941 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.000755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.12.182.126:33777)
0928 03:03:57.001434 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.001441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.001492 (+ 51us) server_negotiation.cc:407] Connection header received
0928 03:03:57.002987 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.002995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.003318 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.003535 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.004701 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.006368 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.007796 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.008634 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.008815 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.012191 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.012198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.012211 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.012242 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:57.015099 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.015844 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.015850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.015852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.016004 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.016399 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.016402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.016404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.016860 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:57.017012 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:57.017579 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 03:03:57.017913 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":475,"thread_start_us":333,"threads_started":1}
I20260928 03:03:57.018966 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:56.999532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33777 (local address 127.0.0.1:34590)
0928 03:03:57.000325 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.000368 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:57.002581 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:57.003566 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:57.003571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:57.004154 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:57.004491 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.004509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.006519 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.006524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.007653 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.007665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.008847 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.008852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.008993 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.009586 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:57.009643 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:57.011977 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:57.015224 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.015235 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.015246 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.015733 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.016105 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.016109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.016113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.016314 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.016988 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:57.016996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:57.017233 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 03:03:57.017479 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 03:03:57.018056 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":98,"thread_start_us":327,"threads_started":1}
I20260928 03:03:57.021840 3395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:03:57.022886 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.022457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50798)
0928 03:03:57.022645 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.022762 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 03:03:57.023185 3346 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
I20260928 03:03:57.024712 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.024362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50804)
0928 03:03:57.024496 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.024576 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 03:03:57.024889 3395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:44709: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab.
Entry for principal kudu/127.12.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab.
I20260928 03:03:57.044725 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.125:44709
--webserver_interface=127.12.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab
--principal=kudu/127.12.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33777,127.12.182.125:44709,127.12.182.124:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:03:57.074211 3395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } attempt: 1
I20260928 03:03:57.075331 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.074858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50812)
0928 03:03:57.075065 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.075190 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 03:03:57.076246 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.075908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50816)
0928 03:03:57.076042 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.076122 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 03:03:57.076406 3395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:44709: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
I20260928 03:03:57.144632 3395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } attempt: 2
I20260928 03:03:57.145741 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.145236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50820)
0928 03:03:57.145464 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.145561 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 03:03:57.146665 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.146326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50822)
0928 03:03:57.146459 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.146544 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 03:03:57.146824 3395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:44709: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
I20260928 03:03:57.238065 3395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } attempt: 3
I20260928 03:03:57.239204 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.238774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50830)
0928 03:03:57.238969 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.239071 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 03:03:57.240159 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.239786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50832)
0928 03:03:57.239931 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.239998 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 03:03:57.240319 3395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:44709: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
W20260928 03:03:57.356186 3402 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:03:57.356518 3402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:57.356616 3402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:57.356737 3402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:57.363816 3402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:57.363905 3402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:57.363986 3402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:57.364050 3402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:57.364133 3402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:57.373800 3402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33777,127.12.182.125:44709,127.12.182.124:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.125.keytab
--principal=kudu/127.12.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.125:44709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:57.375072 3402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:57.376319 3402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 03:03:57.383556 3395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } attempt: 4
I20260928 03:03:57.384644 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.384141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50840)
0928 03:03:57.384429 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.384512 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260928 03:03:57.385684 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.385307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50854)
0928 03:03:57.385449 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.385517 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 03:03:57.386006 3395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.125:44709: Network error: Client connection negotiation failed: client connection to 127.12.182.125:44709: connect: Connection refused (error 111)
W20260928 03:03:57.387609 3407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:57.387583 3410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:57.387542 3408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:57.388691 3402 server_base.cc:1034] running on GCE node
I20260928 03:03:57.389420 3402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:03:57.390818 3402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:57.392033 3402 hybrid_clock.cc:648] HybridClock initialized: now 1790564637391969 us; error 42 us; skew 500 ppm
I20260928 03:03:57.392342 3402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:57.401760 3402 init.cc:377] Logged in from keytab as kudu/127.12.182.125@KRBTEST.COM (short username kudu)
I20260928 03:03:57.403280 3402 webserver.cc:460] Webserver started at http://127.12.182.125:33351/ using document root <none> and password file <none>
I20260928 03:03:57.403987 3402 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:57.404101 3402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:57.404402 3402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:57.407375 3402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/data/instance:
uuid: "1fe52bdc17414ac3a58cbff3d77b4f25"
format_stamp: "Formatted at 2026-09-28 03:03:57 on dist-test-slave-k5rg"
server_key: "2ce936c99791cf98f9e43115ab07e7c0"
server_key_iv: "d9b5287238ee56b122efe6f9a8407bbd"
server_key_version: "encryptionkey@0"
I20260928 03:03:57.408138 3402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/wal/instance:
uuid: "1fe52bdc17414ac3a58cbff3d77b4f25"
format_stamp: "Formatted at 2026-09-28 03:03:57 on dist-test-slave-k5rg"
server_key: "2ce936c99791cf98f9e43115ab07e7c0"
server_key_iv: "d9b5287238ee56b122efe6f9a8407bbd"
server_key_version: "encryptionkey@0"
I20260928 03:03:57.413014 3402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:03:57.416318 3417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:57.417788 3402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:03:57.417958 3402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/wal
uuid: "1fe52bdc17414ac3a58cbff3d77b4f25"
format_stamp: "Formatted at 2026-09-28 03:03:57 on dist-test-slave-k5rg"
server_key: "2ce936c99791cf98f9e43115ab07e7c0"
server_key_iv: "d9b5287238ee56b122efe6f9a8407bbd"
server_key_version: "encryptionkey@0"
I20260928 03:03:57.418157 3402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:57.432408 3402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:57.436393 3402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:57.436651 3402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:57.453449 3402 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.125:44709
I20260928 03:03:57.453471 3468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.125:44709 every 8 connection(s)
I20260928 03:03:57.454811 3402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/data/info.pb
I20260928 03:03:57.457674 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 3402
I20260928 03:03:57.457856 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-1/wal/instance
I20260928 03:03:57.458376 13017 external_mini_cluster.cc:1428] Setting key 06c31ce3bdbbe5b2d3ce1b3f812dcdea
I20260928 03:03:57.458876 3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:57.467281 3469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.182.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.182.124@KRBTEST.COM" created.
I20260928 03:03:57.488577 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.470282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.12.182.126:33777)
0928 03:03:57.470495 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.470501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.471695 (+ 1194us) server_negotiation.cc:407] Connection header received
0928 03:03:57.474076 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.474080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.474150 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.474238 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.476007 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.476682 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.478692 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.479124 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.479220 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.483197 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.483202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.483205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.483243 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:57.485789 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.486556 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.486561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.486563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.486681 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.487038 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.487042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.487044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.487326 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:57.487441 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:57.488117 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 03:03:57.488312 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:03:57.489689 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.470773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33777 (local address 127.0.0.1:34594)
0928 03:03:57.471521 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.471565 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:57.473793 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:57.474463 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:57.474473 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:57.475162 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:57.475836 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.475858 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.476840 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.476845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.478541 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.478555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.479325 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.479331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.479582 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.480241 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:57.480274 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:57.482964 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:57.485929 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.485939 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.485958 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.486445 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.486780 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.486783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.486785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.486949 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.487414 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:57.487420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:57.487808 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 03:03:57.488022 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 03:03:57.488533 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":282,"threads_started":1}
I20260928 03:03:57.491346 3469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
I20260928 03:03:57.505097 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.492079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50866)
0928 03:03:57.492267 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.492285 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:57.492382 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:57.493099 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:57.493107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:57.493516 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:57.493705 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.493723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.495054 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.495059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.496213 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.496227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.496987 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.496997 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.497134 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.497694 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:57.497719 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:57.499576 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:57.502954 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.502958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.502961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.503292 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.503653 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.503656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.503658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.503806 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.504417 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:57.504424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:57.504530 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 03:03:57.504632 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 03:03:57.504834 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 03:03:57.505566 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.492144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.12.182.125:44709)
0928 03:03:57.492708 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.492713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.492739 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:03:57.492780 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.492786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.492953 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.493058 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.493865 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.494921 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.496333 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.496845 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.496933 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.499773 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.499786 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.499802 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.499838 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:57.502842 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.503407 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.503414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.503417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.503568 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.503906 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.503910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.503913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.504310 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:57.504488 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:57.504796 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 03:03:57.505037 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":269,"threads_started":1}
I20260928 03:03:57.507845 3469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:03:57.508818 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.508422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36350)
0928 03:03:57.508566 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.508671 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 03:03:57.509056 3421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
I20260928 03:03:57.510569 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.510247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36352)
0928 03:03:57.510401 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.510473 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 03:03:57.510758 3469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:35903: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab.
Entry for principal kudu/127.12.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab.
I20260928 03:03:57.521883 13017 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmphc7w/build/asan/bin/kudu
/tmp/dist-test-taskmphc7w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.124:35903
--webserver_interface=127.12.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab
--principal=kudu/127.12.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.182.84:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.182.126:33777,127.12.182.125:44709,127.12.182.124:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:03:57.560043 3469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } attempt: 1
I20260928 03:03:57.561213 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.560746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36360)
0928 03:03:57.560942 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.561047 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 03:03:57.562136 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.561800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36370)
0928 03:03:57.561953 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.562024 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 03:03:57.562314 3469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:35903: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
I20260928 03:03:57.630492 3469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } attempt: 2
I20260928 03:03:57.631484 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.631055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36382)
0928 03:03:57.631245 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.631331 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 03:03:57.632414 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.632072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36396)
0928 03:03:57.632231 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.632299 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 03:03:57.632601 3469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:35903: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
I20260928 03:03:57.685284 3395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } attempt: 5
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
I20260928 03:03:57.697965 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.686266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.12.182.125:44709)
0928 03:03:57.686468 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.686473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.686492 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:03:57.686551 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.686555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.686628 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.686723 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.687464 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.688152 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.689333 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.689834 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.689946 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.693102 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.693106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.693110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.693141 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:57.695553 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.696149 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.696153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.696155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.696271 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.696618 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.696623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.696626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.696907 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:57.697036 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:57.697396 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 03:03:57.697631 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:03:57.697990 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.686053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50874)
0928 03:03:57.686263 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.686281 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:57.686404 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:57.686771 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:57.686776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:57.687149 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:57.687312 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.687322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.688291 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.688297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.689176 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.689188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.689972 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.689978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.690154 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.690797 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:57.690814 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:57.692924 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:57.695687 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.695693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.695697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.696039 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.696373 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.696378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.696382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.696519 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.697018 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:57.697023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:57.697156 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 03:03:57.697280 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 03:03:57.697583 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 03:03:57.699082 3395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:03:57.699977 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.699634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36408)
0928 03:03:57.699787 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.699858 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 03:03:57.700783 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.700473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36416)
0928 03:03:57.700615 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.700677 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 03:03:57.700959 3395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:35903: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
I20260928 03:03:57.723790 3469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } attempt: 3
I20260928 03:03:57.724931 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.724475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36422)
0928 03:03:57.724659 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.724774 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 03:03:57.725944 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.725513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36434)
0928 03:03:57.725751 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.725828 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260928 03:03:57.726164 3469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:35903: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
I20260928 03:03:57.752167 3395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } attempt: 1
I20260928 03:03:57.753333 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.752884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36438)
0928 03:03:57.753084 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.753184 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 03:03:57.754302 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.753947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36452)
0928 03:03:57.754109 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.754184 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 03:03:57.754504 3395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:35903: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
I20260928 03:03:57.822705 3395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } attempt: 2
I20260928 03:03:57.823791 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.823309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36454)
0928 03:03:57.823550 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.823638 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 03:03:57.824708 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.824378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36462)
0928 03:03:57.824514 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.824595 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 03:03:57.824887 3395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:35903: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
W20260928 03:03:57.838965 3476 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:03:57.839310 3476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:03:57.839412 3476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:03:57.839525 3476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:03:57.846644 3476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:03:57.846731 3476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:03:57.846817 3476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:03:57.846877 3476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:03:57.846959 3476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:03:57.856563 3476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.182.84:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.182.126:33777,127.12.182.125:44709,127.12.182.124:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/krb5kdc/kudu_127.12.182.124.keytab
--principal=kudu/127.12.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.182.124:35903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:03:57.857787 3476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:03:57.859050 3476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 03:03:57.869359 3469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } attempt: 4
W20260928 03:03:57.870043 3482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:57.870085 3484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:03:57.870622 3481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:03:57.870697 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.870088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36464)
0928 03:03:57.870407 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.870510 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260928 03:03:57.871090 3476 server_base.cc:1034] running on GCE node
I20260928 03:03:57.871768 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.871384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36470)
0928 03:03:57.871540 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.871625 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 03:03:57.871956 3476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260928 03:03:57.872021 3469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.182.124:35903: Network error: Client connection negotiation failed: client connection to 127.12.182.124:35903: connect: Connection refused (error 111)
I20260928 03:03:57.873350 3476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:03:57.874567 3476 hybrid_clock.cc:648] HybridClock initialized: now 1790564637874505 us; error 33 us; skew 500 ppm
I20260928 03:03:57.874884 3476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:03:57.877841 3476 init.cc:377] Logged in from keytab as kudu/127.12.182.124@KRBTEST.COM (short username kudu)
I20260928 03:03:57.879222 3476 webserver.cc:460] Webserver started at http://127.12.182.124:40451/ using document root <none> and password file <none>
I20260928 03:03:57.879886 3476 fs_manager.cc:359] Metadata directory not provided
I20260928 03:03:57.879985 3476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:03:57.880290 3476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:03:57.883277 3476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/data/instance:
uuid: "d10bd972f2b4414b88fc5bd3a04649ee"
format_stamp: "Formatted at 2026-09-28 03:03:57 on dist-test-slave-k5rg"
server_key: "d5af3ccbe0d4d5ace807f8769780216a"
server_key_iv: "4d1f36bc06f376a5609baf3c1d747c50"
server_key_version: "encryptionkey@0"
I20260928 03:03:57.884075 3476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/wal/instance:
uuid: "d10bd972f2b4414b88fc5bd3a04649ee"
format_stamp: "Formatted at 2026-09-28 03:03:57 on dist-test-slave-k5rg"
server_key: "d5af3ccbe0d4d5ace807f8769780216a"
server_key_iv: "4d1f36bc06f376a5609baf3c1d747c50"
server_key_version: "encryptionkey@0"
I20260928 03:03:57.889261 3476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:03:57.892652 3491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:03:57.894032 3476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 03:03:57.894203 3476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/data,/tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/wal
uuid: "d10bd972f2b4414b88fc5bd3a04649ee"
format_stamp: "Formatted at 2026-09-28 03:03:57 on dist-test-slave-k5rg"
server_key: "d5af3ccbe0d4d5ace807f8769780216a"
server_key_iv: "4d1f36bc06f376a5609baf3c1d747c50"
server_key_version: "encryptionkey@0"
I20260928 03:03:57.894421 3476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:03:57.909453 3476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:03:57.913172 3476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:03:57.913426 3476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:03:57.930325 3476 rpc_server.cc:307] RPC server started. Bound to: 127.12.182.124:35903
I20260928 03:03:57.930346 3542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.182.124:35903 every 8 connection(s)
I20260928 03:03:57.931103 3395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } attempt: 3
I20260928 03:03:57.931828 3476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/data/info.pb
I20260928 03:03:57.935803 3543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:03:57.939721 13017 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmphc7w/build/asan/bin/kudu as pid 3476
I20260928 03:03:57.939893 13017 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmphc7w/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564295078766-13017-0/minicluster-data/master-2/wal/instance
I20260928 03:03:57.940295 13017 external_mini_cluster.cc:1428] Setting key ff8516e1cafeff86c22dd25cbdaa0b40
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.126@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
I20260928 03:03:57.944438 3543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:03:57.950035 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.931770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36474)
0928 03:03:57.931987 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.932008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:57.932163 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:57.936027 (+ 3864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:57.936032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:57.936517 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:57.936693 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.936703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.938300 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.938304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.939411 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.939428 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.940855 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.940861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.941051 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.941712 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:57.941731 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:57.943780 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:57.947025 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.947031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.947036 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.947528 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.948067 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.948072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.948077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.948293 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.949149 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:57.949154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:57.949331 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 03:03:57.949429 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 03:03:57.949654 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 03:03:57.952046 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.932898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.12.182.124:35903)
0928 03:03:57.933664 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.933680 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.933754 (+ 74us) server_negotiation.cc:407] Connection header received
0928 03:03:57.935320 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.935339 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.935700 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.936062 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.936850 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.938156 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.939595 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.940690 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.940887 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.944026 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.944033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.944045 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.944082 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:57.946899 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.947683 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.947691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.947698 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.947910 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.948432 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.948437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.948440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.949024 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:57.949837 (+ 813us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:57.950196 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 03:03:57.950682 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":332,"threads_started":1}
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:57.958961 3395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8: Bootstrap starting.
I20260928 03:03:57.962611 3395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:57.963825 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.941303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.12.182.124:35903)
0928 03:03:57.947533 (+ 6230us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.947538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.947561 (+ 23us) server_negotiation.cc:407] Connection header received
0928 03:03:57.947613 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.947617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.947691 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.947803 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.949752 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.950966 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.952166 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.952845 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.952962 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.957641 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.957651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.957656 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.957691 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:57.960861 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.961636 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.961641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.961644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.961802 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.962194 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.962199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.962202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.962481 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:57.962754 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:57.963220 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 03:03:57.963420 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":6080,"thread_start_us":5934,"threads_started":1}
I20260928 03:03:57.963960 3395 log.cc:824] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:57.964891 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.947633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.12.182.126:33777)
0928 03:03:57.947796 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.947802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.948667 (+ 865us) server_negotiation.cc:407] Connection header received
0928 03:03:57.948930 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.948935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.949006 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.949109 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.950602 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.951631 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.953492 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.954077 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.954186 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.958285 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.958291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.958295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.958333 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:57.961440 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.962252 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.962257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.962260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.962436 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.962833 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.962838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.962841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.963255 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:57.963390 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:57.964251 (+ 861us) server_negotiation.cc:299] Negotiation successful
0928 03:03:57.964456 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":63}
I20260928 03:03:57.965775 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.947771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.126:33777 (local address 127.0.0.1:34602)
0928 03:03:57.948539 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.948587 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:57.948778 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:57.949178 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:57.949185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:57.949965 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:57.950422 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.950434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.951852 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.951857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.953327 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.953348 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.954542 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.954549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.954740 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.955379 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:57.955403 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:57.958067 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:57.961562 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.961574 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.961580 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.962094 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.962555 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.962559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.962562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.962721 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.963464 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:57.963471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:57.963868 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 03:03:57.965056 (+ 1188us) client_negotiation.cc:240] Negotiation successful
0928 03:03:57.965386 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":26,"thread_start_us":407,"threads_started":1}
I20260928 03:03:57.967069 3543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:03:57.967211 3395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8: No bootstrap required, opened a new log
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.126@KRBTEST.COM
I20260928 03:03:57.977068 3395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } }
I20260928 03:03:57.977476 3395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03aee46f358f402da7f4acbfbf1c95c8, State: Initialized, Role: FOLLOWER
I20260928 03:03:57.978049 3395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } }
I20260928 03:03:57.979210 3550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } } }
I20260928 03:03:57.979729 3550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:57.980361 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.967919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.12.182.125:44709)
0928 03:03:57.968081 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.968086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.968099 (+ 13us) server_negotiation.cc:407] Connection header received
0928 03:03:57.968144 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.968148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.968199 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.968280 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.968923 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.969554 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.970703 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.971176 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.971285 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.974127 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.974131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.974135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.974166 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:57.977105 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.978010 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.978014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.978016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.978140 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.978753 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.978757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.978761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.979098 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:57.979228 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:57.979768 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 03:03:57.979994 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 03:03:57.980444 3395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:57.981014 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.967655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.125:44709 (local address 127.0.0.1:50890)
0928 03:03:57.967799 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.967815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:57.967902 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:57.968319 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:57.968323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:57.968655 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:57.968778 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.968786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.969728 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.969734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.970569 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.970580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.971315 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.971321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.971500 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.972006 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:57.972022 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:57.973952 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:57.977220 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.977226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.977230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.977900 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.978240 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.978246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.978250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.978406 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.979327 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:57.979332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:57.979482 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 03:03:57.980588 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0928 03:03:57.980800 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 03:03:57.982015 3543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:03:57.983870 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.965688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.12.182.126:33777)
0928 03:03:57.965855 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.965860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.965878 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:03:57.965928 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.965933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.966258 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.966349 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.968005 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.969112 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.970908 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.971595 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.971699 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.976847 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.976853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.976858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.976893 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:57.980338 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.981355 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.981360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.981363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.981525 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.982020 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.982026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.982029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.982418 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:57.982646 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:57.983299 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 03:03:57.983552 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.124@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
I20260928 03:03:57.997295 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.982681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36492)
0928 03:03:57.983032 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:57.983050 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:57.983147 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:57.983923 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:57.983928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:57.984427 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:57.984625 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.984635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.985582 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.985586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.986813 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:57.986828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.987855 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.987861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:57.988027 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.988555 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:57.988576 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:57.991136 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:57.994596 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.994601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.994605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.995094 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.995482 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:57.995487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:57.995490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:57.995627 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:57.996344 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:57.996349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:57.996508 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 03:03:57.996647 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 03:03:57.996932 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
Sep 28 03:03:57 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.182.125@KRBTEST.COM
I20260928 03:03:57.998279 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.983423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.12.182.124:35903)
0928 03:03:57.983604 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.983610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.983626 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:03:57.983669 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.983674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.983751 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.983845 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.984766 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.985464 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.986982 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.987566 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.987682 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.991302 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.991307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.991312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.991347 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:57.994471 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.995234 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.995239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.995242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.995371 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:57.995737 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:57.995742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:57.995745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:57.996128 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:57.996291 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:57.997748 (+ 1457us) server_negotiation.cc:299] Negotiation successful
0928 03:03:57.997955 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:03:58.004426 3543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee: Bootstrap starting.
I20260928 03:03:58.006399 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:57.987410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.12.182.125:44709)
0928 03:03:57.987571 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:57.987576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:57.987793 (+ 217us) server_negotiation.cc:407] Connection header received
0928 03:03:57.988012 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:57.988017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:57.988089 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:57.988154 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:57.990056 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.991114 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.993369 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:57.994018 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:57.994114 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:57.999825 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:57.999831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:57.999836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:57.999877 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:58.003143 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:58.004028 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:58.004034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:58.004038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:58.004199 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:58.004659 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:58.004663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:58.004666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:58.005013 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:58.005255 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:58.005839 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 03:03:58.006073 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:03:58.008766 3543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:58.010201 3543 log.cc:824] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:58.012259 13017 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:03:58.015684 3370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "03aee46f358f402da7f4acbfbf1c95c8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34606
I20260928 03:03:58.016057 3370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 03:03:58.016242 3370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:58.019316 3370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } }
I20260928 03:03:58.021939 3543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee: No bootstrap required, opened a new log
I20260928 03:03:58.022506 3370 leader_election.cc:288] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1fe52bdc17414ac3a58cbff3d77b4f25 (127.12.182.125:44709), d10bd972f2b4414b88fc5bd3a04649ee (127.12.182.124:35903)
W20260928 03:03:58.023437 3561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:03:58.023594 3561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:03:58.009769 3328 master_runner.cc:186] Error getting master registration for 127.12.182.125:44709: OK, instance_id { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" instance_seqno: 1790564637448698 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:03:58.030591 3328 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 03:03:58.032970 3543 raft_consensus.cc:348] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } }
I20260928 03:03:58.033437 3543 raft_consensus.cc:729] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10bd972f2b4414b88fc5bd3a04649ee, State: Initialized, Role: FOLLOWER
I20260928 03:03:58.033968 3543 consensus_queue.cc:260] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } }
I20260928 03:03:58.042747 3566 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } } }
I20260928 03:03:58.043800 3543 sys_catalog.cc:564] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:58.043874 3518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03aee46f358f402da7f4acbfbf1c95c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d10bd972f2b4414b88fc5bd3a04649ee"
I20260928 03:03:58.044405 3566 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:58.044477 3518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:58.048043 3518 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03aee46f358f402da7f4acbfbf1c95c8 in term 1.
I20260928 03:03:58.048820 3347 leader_election.cc:302] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03aee46f358f402da7f4acbfbf1c95c8, d10bd972f2b4414b88fc5bd3a04649ee; no voters:
I20260928 03:03:58.049302 3550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:03:58.062223 3550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [term 1 LEADER]: Becoming Leader. State: Replica: 03aee46f358f402da7f4acbfbf1c95c8, State: Running, Role: LEADER
I20260928 03:03:58.063349 3550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } }
I20260928 03:03:58.081923 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:58.051033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.12.182.126:33777)
0928 03:03:58.051240 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.051247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.056591 (+ 5344us) server_negotiation.cc:407] Connection header received
0928 03:03:58.058021 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.058033 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.058124 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.058201 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.061897 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:58.062984 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:58.064887 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:58.065570 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:58.065690 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:58.072166 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:58.072174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:58.072190 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:58.072252 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:58.075830 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:58.077720 (+ 1890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:58.077726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:58.077729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:58.077895 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:58.078348 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:58.078353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:58.078356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:58.078779 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:58.079035 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:58.080515 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0928 03:03:58.080801 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:03:58.083608 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:58.046767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.12.182.125:44709)
0928 03:03:58.046962 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.046968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.056098 (+ 9130us) server_negotiation.cc:407] Connection header received
0928 03:03:58.058476 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.058482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.058584 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.058656 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.062822 (+ 4166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:58.063896 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:58.066636 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:58.067277 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:58.067372 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:58.074876 (+ 7504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:58.074883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:58.074888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:58.074937 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:58.078407 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:58.081216 (+ 2809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:58.081223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:58.081226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:58.081400 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:58.081840 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:58.081845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:58.081848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:58.082234 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:58.082448 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:58.082991 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 03:03:58.083236 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 03:03:58.092968 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:58.057120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.12.182.124:35903)
0928 03:03:58.057299 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.057306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.059634 (+ 2328us) server_negotiation.cc:407] Connection header received
0928 03:03:58.060523 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.060529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.060610 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.060682 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.067663 (+ 6981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:58.068730 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:58.073092 (+ 4362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:58.073794 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:58.073892 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:58.085849 (+ 11957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:58.085858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:58.085863 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:58.085909 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:58.089549 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:58.090500 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:58.090506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:58.090509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:58.090676 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:58.091151 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:58.091156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:58.091159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:58.091575 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:58.091825 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:58.092375 (+ 550us) server_negotiation.cc:299] Negotiation successful
0928 03:03:58.092600 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 03:03:58.095423 3571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03aee46f358f402da7f4acbfbf1c95c8. Latest consensus state: current_term: 1 leader_uuid: "03aee46f358f402da7f4acbfbf1c95c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } } }
I20260928 03:03:58.095921 3571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:58.097261 3579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:03:58.099543 3579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:03:58.110785 3518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [term 1 FOLLOWER]: Refusing update from remote peer 03aee46f358f402da7f4acbfbf1c95c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:58.112010 3571 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:58.120488 3476 master_runner.cc:186] Error getting master registration for 127.12.182.125:44709: OK, instance_id { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" instance_seqno: 1790564637448698 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:03:58.136163 3476 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 03:03:58.137392 3566 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03aee46f358f402da7f4acbfbf1c95c8. Latest consensus state: current_term: 1 leader_uuid: "03aee46f358f402da7f4acbfbf1c95c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } } }
I20260928 03:03:58.137704 3566 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [sys.catalog]: This master's current role is: FOLLOWER
W20260928 03:03:58.141746 3586 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:03:58.141963 3586 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:03:58.147121 3566 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03aee46f358f402da7f4acbfbf1c95c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } } }
I20260928 03:03:58.147372 3566 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:58.151679 3571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03aee46f358f402da7f4acbfbf1c95c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } } }
I20260928 03:03:58.151968 3571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:03:58.153203 3579 catalog_manager.cc:1347] Generated new cluster ID: 1106bb22f9ad41bfbc113b8fd40b7f44
I20260928 03:03:58.153314 3579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:03:58.171209 3469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } attempt: 5
I20260928 03:03:58.179052 3579 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:03:58.182361 3579 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 03:03:58 dist-test-slave-k5rg krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.182.125@KRBTEST.COM for kudu/127.12.182.124@KRBTEST.COM
I20260928 03:03:58.192639 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:58.171780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.182.124:35903 (local address 127.0.0.1:36508)
0928 03:03:58.172175 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 03:03:58.172197 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:03:58.172314 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:03:58.173183 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:03:58.173188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:03:58.173737 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:03:58.173942 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:58.173957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:58.178276 (+ 4319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:58.178282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:58.179722 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:03:58.179739 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:03:58.180764 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:03:58.180771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:03:58.180986 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:58.181664 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:03:58.181688 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:03:58.184782 (+ 3094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:03:58.189984 (+ 5202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:58.189990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:58.189994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:58.190440 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:58.190851 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:03:58.190856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:03:58.190860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:03:58.191049 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:03:58.191678 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:03:58.191684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:03:58.191840 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 03:03:58.191974 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 03:03:58.192250 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263}
I20260928 03:03:58.193797 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:58.172517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.12.182.124:35903)
0928 03:03:58.172739 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.172744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.172762 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:03:58.172815 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.172820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.172894 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.172986 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.176953 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:58.178139 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:58.179872 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:58.180598 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:58.180721 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:58.184979 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:03:58.184986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:03:58.184992 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:03:58.185034 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:03:58.189853 (+ 4819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:58.190563 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:58.190569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:58.190572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:58.190741 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:03:58.191153 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:03:58.191158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:03:58.191161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:03:58.191504 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:03:58.191749 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:58.192221 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 03:03:58.193031 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 03:03:58.200033 3469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25: Bootstrap starting.
I20260928 03:03:58.205094 3579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8: Generated new TSK 0
I20260928 03:03:58.206153 3579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:03:58.206965 3469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25: Neither blocks nor log segments found. Creating new log.
I20260928 03:03:58.209225 3469 log.cc:824] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25: Log is configured to *not* fsync() on all Append() calls
I20260928 03:03:58.222728 3469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25: No bootstrap required, opened a new log
I20260928 03:03:58.241596 3469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } }
I20260928 03:03:58.242482 3469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe52bdc17414ac3a58cbff3d77b4f25, State: Initialized, Role: FOLLOWER
I20260928 03:03:58.243515 3469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } }
I20260928 03:03:58.255698 3444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03aee46f358f402da7f4acbfbf1c95c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25"
I20260928 03:03:58.256613 3443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:03:58.260154 3443 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [term 1 FOLLOWER]: Refusing update from remote peer 03aee46f358f402da7f4acbfbf1c95c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:03:58.261114 3571 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:03:58.263211 3469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:03:58.325143 3595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03aee46f358f402da7f4acbfbf1c95c8. Latest consensus state: current_term: 1 leader_uuid: "03aee46f358f402da7f4acbfbf1c95c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } } }
I20260928 03:03:58.325089 3594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03aee46f358f402da7f4acbfbf1c95c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } } }
I20260928 03:03:58.325646 3595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:58.325649 3594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:58.326074 3610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03aee46f358f402da7f4acbfbf1c95c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } } }
I20260928 03:03:58.326476 3610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:03:58.326931 3444 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 03aee46f358f402da7f4acbfbf1c95c8 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260928 03:03:58.332895 3399 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.324131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.12.182.126:33777)
0928 03:03:58.324312 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.324318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.329642 (+ 5324us) server_negotiation.cc:407] Connection header received
0928 03:03:58.331032 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.331038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.331119 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.331189 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.332658 (+ 1469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34632: BlockingRecv error: recv got EOF from 127.0.0.1:34632 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 03:03:58.334367 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.320196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.12.182.125:44709)
0928 03:03:58.320407 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.320413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.329177 (+ 8764us) server_negotiation.cc:407] Connection header received
0928 03:03:58.331400 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.331406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.331499 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.331565 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.334162 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50918: BlockingRecv error: recv got EOF from 127.0.0.1:50918 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:03:58.334964 3618 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 03:03:58.340319 3547 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.330101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.12.182.124:35903)
0928 03:03:58.330273 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.330278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.335344 (+ 5066us) server_negotiation.cc:407] Connection header received
0928 03:03:58.335451 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.335456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.335544 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.335655 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.340062 (+ 4407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36520: BlockingRecv error: recv got EOF from 127.0.0.1:36520 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 03:03:58.361665 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.359410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.12.182.125:44709)
0928 03:03:58.359594 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.359611 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.360063 (+ 452us) server_negotiation.cc:407] Connection header received
0928 03:03:58.360294 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.360301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.360441 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.360545 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.361361 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50932: BlockingRecv error: recv got EOF from 127.0.0.1:50932 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 03:03:58.375640 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.373757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.12.182.125:44709)
0928 03:03:58.373935 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.373942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.374237 (+ 295us) server_negotiation.cc:407] Connection header received
0928 03:03:58.374441 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.374448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.374535 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.374640 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.375422 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50948: BlockingRecv error: recv got EOF from 127.0.0.1:50948 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 03:03:58.383494 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.381349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.12.182.125:44709)
0928 03:03:58.381510 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.381516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.382070 (+ 554us) server_negotiation.cc:407] Connection header received
0928 03:03:58.382244 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.382250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.382340 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.382440 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.383241 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50962: BlockingRecv error: recv got EOF from 127.0.0.1:50962 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 03:03:58.393354 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.391266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.12.182.125:44709)
0928 03:03:58.391458 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.391464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.391964 (+ 500us) server_negotiation.cc:407] Connection header received
0928 03:03:58.392209 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.392215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.392302 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.392402 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.393139 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50968: BlockingRecv error: recv got EOF from 127.0.0.1:50968 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260928 03:03:58.406930 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.404675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.12.182.125:44709)
0928 03:03:58.404861 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.404868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.405386 (+ 518us) server_negotiation.cc:407] Connection header received
0928 03:03:58.405677 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.405684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.405778 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.405880 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.406706 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50984: BlockingRecv error: recv got EOF from 127.0.0.1:50984 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 03:03:58.428880 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.426636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.12.182.125:44709)
0928 03:03:58.426836 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.426843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.427392 (+ 549us) server_negotiation.cc:407] Connection header received
0928 03:03:58.427596 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.427603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.427712 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.427820 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.428645 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50986: BlockingRecv error: recv got EOF from 127.0.0.1:50986 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 03:03:58.485920 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.483622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.12.182.125:44709)
0928 03:03:58.483809 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.483815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.484351 (+ 536us) server_negotiation.cc:407] Connection header received
0928 03:03:58.484622 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.484629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.484728 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.484842 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.485670 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51002: BlockingRecv error: recv got EOF from 127.0.0.1:51002 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 03:03:58.555009 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.553092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.12.182.125:44709)
0928 03:03:58.553305 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.553311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.553593 (+ 282us) server_negotiation.cc:407] Connection header received
0928 03:03:58.553806 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.553813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.553913 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.554044 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.554792 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51004: BlockingRecv error: recv got EOF from 127.0.0.1:51004 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:03:58.641271 3571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03aee46f358f402da7f4acbfbf1c95c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03aee46f358f402da7f4acbfbf1c95c8" member_type: VOTER last_known_addr { host: "127.12.182.126" port: 33777 } } peers { permanent_uuid: "1fe52bdc17414ac3a58cbff3d77b4f25" member_type: VOTER last_known_addr { host: "127.12.182.125" port: 44709 } } peers { permanent_uuid: "d10bd972f2b4414b88fc5bd3a04649ee" member_type: VOTER last_known_addr { host: "127.12.182.124" port: 35903 } } }
I20260928 03:03:58.641597 3571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aee46f358f402da7f4acbfbf1c95c8 [sys.catalog]: This master's current role is: LEADER
W20260928 03:03:58.689129 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.687165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.12.182.125:44709)
0928 03:03:58.687428 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.687435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.687686 (+ 251us) server_negotiation.cc:407] Connection header received
0928 03:03:58.687886 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.687890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.687977 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.688076 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.688863 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51008: BlockingRecv error: recv got EOF from 127.0.0.1:51008 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260928 03:03:58.950220 3473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:03:58.948572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.12.182.125:44709)
0928 03:03:58.948809 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:58.948816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:58.949057 (+ 241us) server_negotiation.cc:407] Connection header received
0928 03:03:58.949196 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:58.949201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:58.949278 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:58.949361 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:03:58.949998 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51022: BlockingRecv error: recv got EOF from 127.0.0.1:51022 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 03:03:59.144014 3586 catalog_manager.cc:1259] Loaded cluster ID: 1106bb22f9ad41bfbc113b8fd40b7f44
I20260928 03:03:59.144116 3586 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee: loading cluster ID for follower catalog manager: success
I20260928 03:03:59.146696 3586 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee: acquiring CA information for follower catalog manager: success
I20260928 03:03:59.148375 3586 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d10bd972f2b4414b88fc5bd3a04649ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:03:59.339044 3618 catalog_manager.cc:1259] Loaded cluster ID: 1106bb22f9ad41bfbc113b8fd40b7f44
I20260928 03:03:59.339180 3618 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25: loading cluster ID for follower catalog manager: success
I20260928 03:03:59.341864 3618 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25: acquiring CA information for follower catalog manager: success
I20260928 03:03:59.343570 3618 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1fe52bdc17414ac3a58cbff3d77b4f25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:03:59.473210 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:59.465929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.12.182.125:44709)
0928 03:03:59.466715 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:59.466720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:59.466738 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:03:59.466801 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:59.466806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:59.466962 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:59.467094 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:59.467583 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:59.468755 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:59.470177 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:59.470836 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:59.470980 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:59.471687 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:59.472240 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:59.472378 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:59.472773 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 03:03:59.472861 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":522,"threads_started":1}
I20260928 03:03:59.486904 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:59.479958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.12.182.124:35903)
0928 03:03:59.480456 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:59.480462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:59.480777 (+ 315us) server_negotiation.cc:407] Connection header received
0928 03:03:59.480979 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:59.480984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:59.481185 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:59.481375 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:59.481842 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:59.482939 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:59.484351 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:59.484998 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:59.485126 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:59.485621 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:59.486300 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:59.486449 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:59.486497 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 03:03:59.486582 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":211,"threads_started":1}
I20260928 03:03:59.500613 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:59.493793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.12.182.125:44709)
0928 03:03:59.494013 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:59.494019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:59.494147 (+ 128us) server_negotiation.cc:407] Connection header received
0928 03:03:59.494388 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:59.494394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:59.494486 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:59.494587 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:59.495006 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:59.496306 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:59.498496 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:59.499219 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:59.499348 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:59.499833 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:59.499975 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:59.500066 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:59.500214 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 03:03:59.500297 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 03:03:59.507941 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:59.501761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.12.182.124:35903)
0928 03:03:59.501946 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:59.501952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:59.502236 (+ 284us) server_negotiation.cc:407] Connection header received
0928 03:03:59.502484 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:59.502490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:59.502574 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:59.502664 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:59.503134 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:59.504227 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:59.506100 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:59.506711 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:59.506829 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:59.507251 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:59.507402 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:59.507483 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:59.507600 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 03:03:59.507675 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:03:59.510887 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:03:59.493491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.12.182.126:33777)
0928 03:03:59.505852 (+ 12361us) server_negotiation.cc:206] Beginning negotiation
0928 03:03:59.505859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:03:59.505878 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:03:59.505988 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:03:59.505993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:03:59.506200 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:03:59.506349 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:03:59.506729 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:59.507599 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:59.509012 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:03:59.509404 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:03:59.509507 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:03:59.509984 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:03:59.510321 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:03:59.510414 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 03:03:59.510538 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 03:03:59.510624 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12240,"thread_start_us":12970,"threads_started":1}
I20260928 03:03:59.515947 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 3328
I20260928 03:03:59.516783 3390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:59.624898 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 3328
W20260928 03:03:59.642148 3494 connection.cc:535] server connection from 127.0.0.1:36474 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 03:03:59.642763 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 3402
I20260928 03:03:59.643764 3464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:59.757128 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 3402
I20260928 03:03:59.773058 13017 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmphc7w/build/asan/bin/kudu pid 3476
I20260928 03:03:59.774134 3538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:03:59.879305 13017 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmphc7w/build/asan/bin/kudu with pid 3476
2026-09-28T03:03:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4575 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9368 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (344766 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST